2026-03-20 15:34:16.508582 | Job console starting... 2026-03-20 15:34:16.529168 | Updating repositories 2026-03-20 15:34:16.628273 | Preparing job workspace 2026-03-20 15:34:28.624025 | Running Ansible setup... 2026-03-20 15:34:34.759050 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/base-minimal/pre.yaml@master] 2026-03-20 15:34:35.351818 | 2026-03-20 15:34:35.351989 | PLAY [localhost] 2026-03-20 15:34:35.361264 | 2026-03-20 15:34:35.361334 | TASK [Gathering Facts] 2026-03-20 15:34:36.334982 | localhost | ok 2026-03-20 15:34:36.364455 | 2026-03-20 15:34:36.364653 | TASK [log-inventory : Ensure Zuul Ansible directory exists] 2026-03-20 15:34:36.818826 | localhost -> localhost | changed 2026-03-20 15:34:36.829861 | 2026-03-20 15:34:36.829998 | TASK [log-inventory : Copy ansible inventory to logs dir] 2026-03-20 15:34:37.721718 | localhost -> localhost | changed 2026-03-20 15:34:37.739188 | 2026-03-20 15:34:37.739321 | TASK [Setup log path fact] 2026-03-20 15:34:37.763216 | localhost | ok 2026-03-20 15:34:37.787187 | 2026-03-20 15:34:37.787322 | TASK [set-zuul-log-path-fact : Set log path for a build] 2026-03-20 15:34:37.830878 | localhost | ok 2026-03-20 15:34:37.847320 | 2026-03-20 15:34:37.847456 | TASK [emit-job-header : Print job information] 2026-03-20 15:34:37.901522 | # Job Information 2026-03-20 15:34:37.901788 | Ansible Version: 2.15.12 2026-03-20 15:34:37.901846 | Job: watcher-operator-kuttl 2026-03-20 15:34:37.901888 | Pipeline: github-check 2026-03-20 15:34:37.901927 | Executor: ze02.softwarefactory-project.io 2026-03-20 15:34:37.901964 | Triggered by: https://github.com/openstack-k8s-operators/watcher-operator/pull/354 2026-03-20 15:34:37.902003 | Log URL (when completed): https://logserver.rdoproject.org/7ff/rdoproject.org/7ffee4ddf4ea401db94c9ba86e43148b/ 2026-03-20 15:34:37.902042 | Event ID: da2ff540-246c-11f1-8e6c-12c77a3bf6cf 2026-03-20 15:34:37.911320 | 2026-03-20 15:34:37.911448 | LOOP [emit-job-header : Print node information] 2026-03-20 15:34:38.035166 | localhost | ok: 2026-03-20 15:34:38.035636 | localhost | # Node Information 2026-03-20 15:34:38.035767 | localhost | Inventory Hostname: controller 2026-03-20 15:34:38.035838 | localhost | Hostname: np0005653376 2026-03-20 15:34:38.035897 | localhost | Username: zuul 2026-03-20 15:34:38.035958 | localhost | Distro: CentOS 9 2026-03-20 15:34:38.036013 | localhost | Provider: vexxhost-nodepool-tripleo 2026-03-20 15:34:38.036064 | localhost | Region: RegionOne 2026-03-20 15:34:38.036114 | localhost | Label: cloud-centos-9-stream-tripleo-vexxhost-medium 2026-03-20 15:34:38.036163 | localhost | Product Name: OpenStack Nova 2026-03-20 15:34:38.036216 | localhost | Interface IP: 38.102.83.159 2026-03-20 15:34:38.118986 | localhost | ok: 2026-03-20 15:34:38.119198 | localhost | # Node Information 2026-03-20 15:34:38.119283 | localhost | Inventory Hostname: crc 2026-03-20 15:34:38.119351 | localhost | Hostname: crc 2026-03-20 15:34:38.119411 | localhost | Username: core 2026-03-20 15:34:38.119469 | localhost | Distro: RedHat 4.18 2026-03-20 15:34:38.119514 | localhost | Provider: vexxhost-nodepool-tripleo 2026-03-20 15:34:38.119554 | localhost | Region: RegionOne 2026-03-20 15:34:38.119592 | localhost | Label: crc-cloud-ocp-4-18-1-3xl 2026-03-20 15:34:38.119628 | localhost | Product Name: OpenStack Nova 2026-03-20 15:34:38.119699 | localhost | Interface IP: 38.102.83.181 2026-03-20 15:34:38.161464 | 2026-03-20 15:34:38.161615 | PLAY [all] 2026-03-20 15:34:38.176269 | 2026-03-20 15:34:38.176396 | TASK [Gather network facts] 2026-03-20 15:34:38.571419 | controller | ok 2026-03-20 15:34:38.881171 | crc | ok 2026-03-20 15:34:38.928743 | 2026-03-20 15:34:38.928843 | TASK [include_role : start-zuul-console] 2026-03-20 15:34:38.962650 | controller | ok 2026-03-20 15:34:38.967774 | crc | ok 2026-03-20 15:34:38.985561 | 2026-03-20 15:34:38.985632 | TASK [start-zuul-console : Start zuul_console daemon.] 2026-03-20 15:34:39.340485 | controller | ok 2026-03-20 15:34:39.397276 | crc | ok 2026-03-20 15:34:39.427416 | 2026-03-20 15:34:39.427542 | TASK [include_role : add-build-sshkey] 2026-03-20 15:34:39.464016 | controller | ok 2026-03-20 15:34:39.471145 | crc | ok 2026-03-20 15:34:39.496867 | 2026-03-20 15:34:39.496962 | TASK [add-build-sshkey : Check to see if ssh key was already created for this build] 2026-03-20 15:34:39.779117 | controller -> localhost | ok 2026-03-20 15:34:39.827603 | 2026-03-20 15:34:39.827824 | TASK [add-build-sshkey : Create a new key in workspace based on build UUID] 2026-03-20 15:34:39.864856 | controller | ok 2026-03-20 15:34:39.882574 | crc | ok 2026-03-20 15:34:39.907029 | controller | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/trusted/project_1/opendev.org/zuul/zuul-jobs/roles/add-build-sshkey/tasks/create-key-and-replace.yaml 2026-03-20 15:34:39.907101 | crc | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/trusted/project_1/opendev.org/zuul/zuul-jobs/roles/add-build-sshkey/tasks/create-key-and-replace.yaml 2026-03-20 15:34:39.915502 | 2026-03-20 15:34:39.915586 | TASK [add-build-sshkey : Create Temp SSH key] 2026-03-20 15:34:40.429922 | controller -> localhost | Generating public/private rsa key pair. 2026-03-20 15:34:40.430319 | controller -> localhost | Your identification has been saved in /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/work/7ffee4ddf4ea401db94c9ba86e43148b_id_rsa. 2026-03-20 15:34:40.430399 | controller -> localhost | Your public key has been saved in /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/work/7ffee4ddf4ea401db94c9ba86e43148b_id_rsa.pub. 2026-03-20 15:34:40.430472 | controller -> localhost | The key fingerprint is: 2026-03-20 15:34:40.430524 | controller -> localhost | SHA256:ENPzC1M1wSNfkars1ppFc61+YESzYGeZIkDPPnZuY5M zuul-build-sshkey 2026-03-20 15:34:40.430572 | controller -> localhost | The key's randomart image is: 2026-03-20 15:34:40.430616 | controller -> localhost | +---[RSA 3072]----+ 2026-03-20 15:34:40.430658 | controller -> localhost | | ooo. o+..= | 2026-03-20 15:34:40.430762 | controller -> localhost | | ooo+ *.X | 2026-03-20 15:34:40.430812 | controller -> localhost | | . +o= X o | 2026-03-20 15:34:40.430856 | controller -> localhost | | .o.. o o. | 2026-03-20 15:34:40.430896 | controller -> localhost | | S++o+.. .| 2026-03-20 15:34:40.430940 | controller -> localhost | | .+= +o. | 2026-03-20 15:34:40.430981 | controller -> localhost | | . .E... | 2026-03-20 15:34:40.431022 | controller -> localhost | | o=.+ .| 2026-03-20 15:34:40.431063 | controller -> localhost | | .o. .. | 2026-03-20 15:34:40.431102 | controller -> localhost | +----[SHA256]-----+ 2026-03-20 15:34:40.431206 | controller -> localhost | ok: Runtime: 0:00:00.082195 2026-03-20 15:34:40.443708 | 2026-03-20 15:34:40.443844 | TASK [add-build-sshkey : Remote setup ssh keys (linux)] 2026-03-20 15:34:40.488874 | controller | ok 2026-03-20 15:34:40.506444 | crc | ok 2026-03-20 15:34:40.526210 | controller | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/trusted/project_1/opendev.org/zuul/zuul-jobs/roles/add-build-sshkey/tasks/remote-linux.yaml 2026-03-20 15:34:40.526279 | crc | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/trusted/project_1/opendev.org/zuul/zuul-jobs/roles/add-build-sshkey/tasks/remote-linux.yaml 2026-03-20 15:34:40.541307 | 2026-03-20 15:34:40.541367 | TASK [add-build-sshkey : Remove previously added zuul-build-sshkey] 2026-03-20 15:34:40.566127 | controller | skipping: Conditional result was False 2026-03-20 15:34:40.566743 | crc | skipping: Conditional result was False 2026-03-20 15:34:40.572816 | 2026-03-20 15:34:40.572883 | TASK [add-build-sshkey : Enable access via build key on all nodes] 2026-03-20 15:34:40.922642 | controller | changed 2026-03-20 15:34:41.072716 | crc | changed 2026-03-20 15:34:41.079096 | 2026-03-20 15:34:41.079175 | TASK [add-build-sshkey : Make sure user has a .ssh] 2026-03-20 15:34:41.347550 | controller | ok 2026-03-20 15:34:41.411298 | crc | ok 2026-03-20 15:34:41.422047 | 2026-03-20 15:34:41.422202 | TASK [add-build-sshkey : Install build private key as SSH key on all nodes] 2026-03-20 15:34:42.015028 | controller | changed 2026-03-20 15:34:42.303493 | crc | changed 2026-03-20 15:34:42.315321 | 2026-03-20 15:34:42.315476 | TASK [add-build-sshkey : Install build public key as SSH key on all nodes] 2026-03-20 15:34:42.937080 | controller | changed 2026-03-20 15:34:43.278428 | crc | changed 2026-03-20 15:34:43.289381 | 2026-03-20 15:34:43.289521 | TASK [add-build-sshkey : Remote setup ssh keys (windows)] 2026-03-20 15:34:43.321090 | controller | skipping: Conditional result was False 2026-03-20 15:34:43.343157 | crc | skipping: Conditional result was False 2026-03-20 15:34:43.350915 | 2026-03-20 15:34:43.351001 | TASK [remove-zuul-sshkey : Remove master key from local agent] 2026-03-20 15:34:43.711595 | controller -> localhost | changed 2026-03-20 15:34:43.745735 | 2026-03-20 15:34:43.745886 | TASK [add-build-sshkey : Add back temp key] 2026-03-20 15:34:44.089982 | controller -> localhost | Identity added: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/work/7ffee4ddf4ea401db94c9ba86e43148b_id_rsa (zuul-build-sshkey) 2026-03-20 15:34:44.090327 | controller -> localhost | ok: Runtime: 0:00:00.017010 2026-03-20 15:34:44.102512 | 2026-03-20 15:34:44.102646 | TASK [add-build-sshkey : Verify we can still SSH to all nodes] 2026-03-20 15:34:44.468968 | controller | ok 2026-03-20 15:34:44.568249 | crc | ok 2026-03-20 15:34:44.574819 | 2026-03-20 15:34:44.574906 | TASK [add-build-sshkey : Verify we can still SSH to all nodes (windows)] 2026-03-20 15:34:44.606533 | controller | skipping: Conditional result was False 2026-03-20 15:34:44.617479 | crc | skipping: Conditional result was False 2026-03-20 15:34:44.638434 | 2026-03-20 15:34:44.638542 | TASK [include_role : validate-host] 2026-03-20 15:34:44.675104 | controller | ok 2026-03-20 15:34:44.692588 | crc | ok 2026-03-20 15:34:44.740613 | 2026-03-20 15:34:44.740858 | TASK [validate-host : Define zuul_info_dir fact] 2026-03-20 15:34:44.795443 | controller | ok 2026-03-20 15:34:44.807360 | crc | ok 2026-03-20 15:34:44.815419 | 2026-03-20 15:34:44.815505 | TASK [validate-host : Ensure Zuul Ansible directory exists] 2026-03-20 15:34:45.065871 | controller -> localhost | ok 2026-03-20 15:34:45.078583 | 2026-03-20 15:34:45.078783 | TASK [validate-host : Collect information about the host] 2026-03-20 15:34:45.770257 | controller | ok 2026-03-20 15:34:46.146980 | crc | ok 2026-03-20 15:34:46.165199 | 2026-03-20 15:34:46.165364 | TASK [validate-host : Sanitize hostname] 2026-03-20 15:34:46.235908 | controller | ok 2026-03-20 15:34:46.259008 | crc | ok 2026-03-20 15:34:46.276728 | 2026-03-20 15:34:46.276923 | TASK [validate-host : Write out all ansible variables/facts known for each host] 2026-03-20 15:34:46.753246 | controller -> localhost | changed 2026-03-20 15:34:46.795253 | crc -> localhost | changed 2026-03-20 15:34:46.800655 | 2026-03-20 15:34:46.800734 | TASK [validate-host : Collect information about zuul worker] 2026-03-20 15:34:47.179995 | controller | ok 2026-03-20 15:34:47.316944 | crc | ok 2026-03-20 15:34:47.326738 | 2026-03-20 15:34:47.326865 | TASK [validate-host : Write out all zuul information for each host] 2026-03-20 15:34:47.801826 | crc -> localhost | changed 2026-03-20 15:34:47.833644 | controller -> localhost | changed 2026-03-20 15:34:47.847772 | 2026-03-20 15:34:47.847837 | TASK [include_role : prepare-workspace-openshift] 2026-03-20 15:34:47.862520 | controller | skipping: Conditional result was False 2026-03-20 15:34:47.868078 | 2026-03-20 15:34:47.868186 | TASK [include_role : remove-zuul-sshkey] 2026-03-20 15:34:47.892186 | controller | skipping: Conditional result was False 2026-03-20 15:34:47.897628 | 2026-03-20 15:34:47.897707 | LOOP [ensure-output-dirs : Empty Zuul Output directories by removing them] 2026-03-20 15:34:48.161002 | controller | ok: "logs" 2026-03-20 15:34:48.161435 | controller | ok: All items complete 2026-03-20 15:34:48.161495 | 2026-03-20 15:34:48.217716 | crc | ok: "logs" 2026-03-20 15:34:48.337046 | controller | ok: "artifacts" 2026-03-20 15:34:48.497810 | crc | ok: "artifacts" 2026-03-20 15:34:48.549906 | controller | ok: "docs" 2026-03-20 15:34:48.741162 | crc | ok: "docs" 2026-03-20 15:34:48.754004 | 2026-03-20 15:34:48.754262 | LOOP [ensure-output-dirs : Ensure Zuul Output directories exist] 2026-03-20 15:34:49.039321 | controller | changed: "logs" 2026-03-20 15:34:49.039782 | controller | ok: All items complete 2026-03-20 15:34:49.039855 | 2026-03-20 15:34:49.130877 | crc | changed: "logs" 2026-03-20 15:34:49.237046 | controller | changed: "artifacts" 2026-03-20 15:34:49.401187 | crc | changed: "artifacts" 2026-03-20 15:34:49.448913 | controller | changed: "docs" 2026-03-20 15:34:49.670183 | crc | changed: "docs" 2026-03-20 15:34:49.770245 | 2026-03-20 15:34:49.770359 | PLAY RECAP 2026-03-20 15:34:49.770401 | controller | ok: 22 changed: 9 unreachable: 0 failed: 0 skipped: 5 rescued: 0 ignored: 0 2026-03-20 15:34:49.770426 | crc | ok: 17 changed: 6 unreachable: 0 failed: 0 skipped: 3 rescued: 0 ignored: 0 2026-03-20 15:34:49.770448 | localhost | ok: 6 changed: 2 unreachable: 0 failed: 0 skipped: 0 rescued: 0 ignored: 0 2026-03-20 15:34:49.770464 | 2026-03-20 15:34:49.872462 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/base-minimal/pre.yaml@master] 2026-03-20 15:34:49.878511 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/ci-framework-rdo-base/pre.yaml@master] 2026-03-20 15:34:50.469929 | 2026-03-20 15:34:50.470077 | PLAY [all] 2026-03-20 15:34:50.502505 | 2026-03-20 15:34:50.502588 | TASK [mirror-info-fork : Set mirror_fqdn if not defined] 2026-03-20 15:34:50.562773 | controller | ok 2026-03-20 15:34:50.579401 | crc | ok 2026-03-20 15:34:50.584104 | 2026-03-20 15:34:50.584177 | TASK [mirror-info-fork : Create /etc/ci] 2026-03-20 15:34:50.985983 | controller | changed 2026-03-20 15:34:51.043823 | crc | changed 2026-03-20 15:34:51.056004 | 2026-03-20 15:34:51.056144 | TASK [mirror-info-fork : Install ci_mirror script] 2026-03-20 15:34:52.088461 | controller | changed 2026-03-20 15:34:52.168860 | crc | changed 2026-03-20 15:34:52.189117 | 2026-03-20 15:34:52.189194 | LOOP [add-authorized-keys : Enable access via build key on all nodes] 2026-03-20 15:34:52.613103 | controller | changed: 2026-03-20 15:34:52.613645 | controller | { 2026-03-20 15:34:52.613822 | controller | "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA4Z/c9osaGGtU6X8fgELwfj/yayRurfcKA0HMFfdpPxev2dbwljysMuzoVp4OZmW1gvGtyYPSNRvnzgsaabPNKNo2ym5NToCP6UM+KSe93aln4BcM/24mXChYAbXJQ5Bqq/pIzsGs/pKetQN+vwvMxLOwTvpcsCJBXaa981RKML6xj9l/UZ7IIq1HSEKMvPLxZMWdu0Ut8DkCd5F4nOw9Wgml2uYpDCj5LLCrQQ9ChdOMz8hz6SighhNlRpPkvPaet3OXxr/ytFMu7j7vv06CaEnuMMiY2aTWN1Imin9eHAylIqFHta/3gFfQSWt9jXM7owkBLKL7ATzhaAn+fjNupw== arxcruz@redhat.com" 2026-03-20 15:34:52.613904 | controller | } 2026-03-20 15:34:52.719495 | crc | changed: 2026-03-20 15:34:52.719609 | crc | { 2026-03-20 15:34:52.719648 | crc | "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA4Z/c9osaGGtU6X8fgELwfj/yayRurfcKA0HMFfdpPxev2dbwljysMuzoVp4OZmW1gvGtyYPSNRvnzgsaabPNKNo2ym5NToCP6UM+KSe93aln4BcM/24mXChYAbXJQ5Bqq/pIzsGs/pKetQN+vwvMxLOwTvpcsCJBXaa981RKML6xj9l/UZ7IIq1HSEKMvPLxZMWdu0Ut8DkCd5F4nOw9Wgml2uYpDCj5LLCrQQ9ChdOMz8hz6SighhNlRpPkvPaet3OXxr/ytFMu7j7vv06CaEnuMMiY2aTWN1Imin9eHAylIqFHta/3gFfQSWt9jXM7owkBLKL7ATzhaAn+fjNupw== arxcruz@redhat.com" 2026-03-20 15:34:52.719698 | crc | } 2026-03-20 15:34:52.832399 | controller | changed: 2026-03-20 15:34:52.832548 | controller | { 2026-03-20 15:34:52.832596 | controller | "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDS4Fn6k4deCnIlOtLWqZJyksbepjQt04j8Ed8CGx9EKkj0fKiAxiI4TadXQYPuNHMixZy4Nevjb6aDhL5Z906TfvNHKUrjrG7G26a0k8vdc61NEQ7FmcGMWRLwwc6ReDO7lFpzYKBMk4YqfWgBuGU/K6WLKiVW2cVvwIuGIaYrE1OiiX0iVUUk7KApXlDJMXn7qjSYynfO4mF629NIp8FJal38+Kv+HA+0QkE5Y2xXnzD4Lar5+keymiCHRntPppXHeLIRzbt0gxC7v3L72hpQ3BTBEzwHpeS8KY+SX1y5lRMN45thCHfJqGmARJREDjBvWG8JXOPmVIKQtZmVcD5b mandreou@redhat.com" 2026-03-20 15:34:52.832636 | controller | } 2026-03-20 15:34:53.056559 | controller | changed: 2026-03-20 15:34:53.056716 | controller | { 2026-03-20 15:34:53.056767 | controller | "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9MiLfy30deHA7xPOAlew5qUq3UP2gmRMYJi8PtkjFB20/DKeWwWNnkZPqP9AayruRoo51SIiVg870gbZE2jYl+Ncx/FYDe56JeC3ySZsXoAVkC9bP7gkOGqOmJjirvAgPMI7bogVz8i+66Q4Ar7OKTp3762G4IuWPPEg4ce4Y7lx9qWocZapHYq4cYKMxrOZ7SEbFSATBbe2bPZAPKTw8do/Eny+Hq/LkHFhIeyra6cqTFQYShr+zPln0Cr+ro/pDX3bB+1ubFgTpjpkkkQsLhDfR6cCdCWM2lgnS3BTtYj5Ct9/JRPR5YOphqZz+uB+OEu2IL68hmU9vNTth1KeX rlandy@redhat.com" 2026-03-20 15:34:53.056823 | controller | } 2026-03-20 15:34:53.133754 | crc | changed: 2026-03-20 15:34:53.133894 | crc | { 2026-03-20 15:34:53.133946 | crc | "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDS4Fn6k4deCnIlOtLWqZJyksbepjQt04j8Ed8CGx9EKkj0fKiAxiI4TadXQYPuNHMixZy4Nevjb6aDhL5Z906TfvNHKUrjrG7G26a0k8vdc61NEQ7FmcGMWRLwwc6ReDO7lFpzYKBMk4YqfWgBuGU/K6WLKiVW2cVvwIuGIaYrE1OiiX0iVUUk7KApXlDJMXn7qjSYynfO4mF629NIp8FJal38+Kv+HA+0QkE5Y2xXnzD4Lar5+keymiCHRntPppXHeLIRzbt0gxC7v3L72hpQ3BTBEzwHpeS8KY+SX1y5lRMN45thCHfJqGmARJREDjBvWG8JXOPmVIKQtZmVcD5b mandreou@redhat.com" 2026-03-20 15:34:53.133988 | crc | } 2026-03-20 15:34:53.297281 | controller | changed: 2026-03-20 15:34:53.297455 | controller | { 2026-03-20 15:34:53.297525 | controller | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFCbgz8gdERiJlk2IKOtkjQxEXejrio6ZYMJAVJYpOIp raukadah@gmail.com" 2026-03-20 15:34:53.297583 | controller | } 2026-03-20 15:34:53.537947 | controller | changed: 2026-03-20 15:34:53.538133 | controller | { 2026-03-20 15:34:53.538204 | controller | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBqb3Q/9uDf4LmihQ7xeJ9gA/STIQUFPSfyyV0m8AoQi bshewale@redhat.com" 2026-03-20 15:34:53.538263 | controller | } 2026-03-20 15:34:53.553495 | crc | changed: 2026-03-20 15:34:53.553741 | crc | { 2026-03-20 15:34:53.553831 | crc | "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9MiLfy30deHA7xPOAlew5qUq3UP2gmRMYJi8PtkjFB20/DKeWwWNnkZPqP9AayruRoo51SIiVg870gbZE2jYl+Ncx/FYDe56JeC3ySZsXoAVkC9bP7gkOGqOmJjirvAgPMI7bogVz8i+66Q4Ar7OKTp3762G4IuWPPEg4ce4Y7lx9qWocZapHYq4cYKMxrOZ7SEbFSATBbe2bPZAPKTw8do/Eny+Hq/LkHFhIeyra6cqTFQYShr+zPln0Cr+ro/pDX3bB+1ubFgTpjpkkkQsLhDfR6cCdCWM2lgnS3BTtYj5Ct9/JRPR5YOphqZz+uB+OEu2IL68hmU9vNTth1KeX rlandy@redhat.com" 2026-03-20 15:34:53.553906 | crc | } 2026-03-20 15:34:53.784201 | controller | changed: 2026-03-20 15:34:53.784345 | controller | { 2026-03-20 15:34:53.784395 | controller | "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0I8QqQx0Az2ysJt2JuffucLijhBqnsXKEIx5GyHwxVULROa8VtNFXUDH6ZKZavhiMcmfHB2+TBTda+lDP4FldYj06dGmzCY+IYGa+uDRdxHNGYjvCfLFcmLlzRK6fNbTcui+KlUFUdKe0fb9CRoGKyhlJD5GRkM1Dv+Yb6Bj+RNnmm1fVGYxzmrD2utvffYEb0SZGWxq2R9gefx1q/3wCGjeqvufEV+AskPhVGc5T7t9eyZ4qmslkLh1/nMuaIBFcr9AUACRajsvk6mXrAN1g3HlBf2gQlhi1UEyfbqIQvzzFtsbLDlSum/KmKjy818GzvWjERfQ0VkGzCd9bSLVL dviroel@redhat.com" 2026-03-20 15:34:53.784437 | controller | } 2026-03-20 15:34:53.891250 | crc | changed: 2026-03-20 15:34:53.891446 | crc | { 2026-03-20 15:34:53.891501 | crc | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFCbgz8gdERiJlk2IKOtkjQxEXejrio6ZYMJAVJYpOIp raukadah@gmail.com" 2026-03-20 15:34:53.891558 | crc | } 2026-03-20 15:34:54.008338 | controller | changed: 2026-03-20 15:34:54.008480 | controller | { 2026-03-20 15:34:54.008529 | controller | "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLOQd4ZLtkZXQGY6UwAr/06ppWQK4fDO3HaqxPk98csyOCBXsliSKK39Bso828+5srIXiW7aI6aC9P5mwi4mUZlGPfJlQbfrcGvY+b/SocuvaGK+1RrHLoJCT52LBhwgrzlXio2jeksZeein8iaTrhsPrOAs7KggIL/rB9hEiB3NaOPWhhoCP4vlW6MEMExGcqB/1FVxXFBPnLkEyW0Lk7ycVflZl2ocRxbfjZi0+tI1Wlinp8PvSQSc/WVrAcDgKjc/mB4ODPOyYy3G8FHgfMsrXSDEyjBKgLKMsdCrAUcqJQWjkqXleXSYOV4q3pzL+9umK+q/e3P/bIoSFQzmJKTU1eDfuvPXmow9F5H54fii/Da7ezlMJ+wPGHJrRAkmzvMbALy7xwswLhZMkOGNtRcPqaKYRmIBKpw3o6bCTtcNUHOtOQnzwY8JzrM2eBWJBXAANYw+9/ho80JIiwhg29CFNpVBuHbql2YxJQNrnl90guN65rYNpDxdIluweyUf8= anbanerj@kaermorhen" 2026-03-20 15:34:54.008569 | controller | } 2026-03-20 15:34:54.229835 | crc | changed: 2026-03-20 15:34:54.229984 | crc | { 2026-03-20 15:34:54.230032 | crc | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBqb3Q/9uDf4LmihQ7xeJ9gA/STIQUFPSfyyV0m8AoQi bshewale@redhat.com" 2026-03-20 15:34:54.230087 | crc | } 2026-03-20 15:34:54.240233 | controller | changed: 2026-03-20 15:34:54.240331 | controller | { 2026-03-20 15:34:54.240381 | controller | "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3VwV8Im9kRm49lt3tM36hj4Zv27FxGo4C1Q/0jqhzFmHY7RHbmeRr8ObhwWoHjXSozKWg8FL5ER0z3hTwL0W6lez3sL7hUaCmSuZmG5Hnl3x4vTSxDI9JZ/Y65rtYiiWQo2fC5xJhU/4+0e5e/pseCm8cKRSu+SaxhO+sd6FDojA2x1BzOzKiQRDy/1zWGp/cZkxcEuB1wHI5LMzN03c67vmbu+fhZRAUO4dQkvcnj2LrhQtpa+ytvnSjr8icMDosf1OsbSffwZFyHB/hfWGAfe0eIeSA2XPraxiPknXxiPKx2MJsaUTYbsZcm3EjFdHBBMumw5rBI74zLrMRvCO9GwBEmGT4rFng1nP+yw5DB8sn2zqpOsPg1LYRwCPOUveC13P6pgsZZPh812e8v5EKnETct+5XI3dVpdw6CnNiLwAyVAF15DJvBGT/u1k0Myg/bQn+Gv9k2MSj6LvQmf6WbZu2Wgjm30z3FyCneBqTL7mLF19YXzeC0ufHz5pnO1E= dasm@fedora" 2026-03-20 15:34:54.240423 | controller | } 2026-03-20 15:34:54.424275 | controller | changed: 2026-03-20 15:34:54.424439 | controller | { 2026-03-20 15:34:54.424501 | controller | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHUnwjB20UKmsSed9X73eGNV5AOEFccQ3NYrRW776pEk cjeanner" 2026-03-20 15:34:54.424550 | controller | } 2026-03-20 15:34:54.589627 | crc | changed: 2026-03-20 15:34:54.589786 | crc | { 2026-03-20 15:34:54.589835 | crc | "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0I8QqQx0Az2ysJt2JuffucLijhBqnsXKEIx5GyHwxVULROa8VtNFXUDH6ZKZavhiMcmfHB2+TBTda+lDP4FldYj06dGmzCY+IYGa+uDRdxHNGYjvCfLFcmLlzRK6fNbTcui+KlUFUdKe0fb9CRoGKyhlJD5GRkM1Dv+Yb6Bj+RNnmm1fVGYxzmrD2utvffYEb0SZGWxq2R9gefx1q/3wCGjeqvufEV+AskPhVGc5T7t9eyZ4qmslkLh1/nMuaIBFcr9AUACRajsvk6mXrAN1g3HlBf2gQlhi1UEyfbqIQvzzFtsbLDlSum/KmKjy818GzvWjERfQ0VkGzCd9bSLVL dviroel@redhat.com" 2026-03-20 15:34:54.589877 | crc | } 2026-03-20 15:34:54.692146 | controller | changed: 2026-03-20 15:34:54.692506 | controller | { 2026-03-20 15:34:54.692567 | controller | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDercCMGn8rW1C4P67tHgtflPdTeXlpyUJYH+6XDd2lR jgilaber@redhat.com" 2026-03-20 15:34:54.692603 | controller | } 2026-03-20 15:34:54.969213 | crc | changed: 2026-03-20 15:34:54.969387 | crc | { 2026-03-20 15:34:54.969442 | crc | "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLOQd4ZLtkZXQGY6UwAr/06ppWQK4fDO3HaqxPk98csyOCBXsliSKK39Bso828+5srIXiW7aI6aC9P5mwi4mUZlGPfJlQbfrcGvY+b/SocuvaGK+1RrHLoJCT52LBhwgrzlXio2jeksZeein8iaTrhsPrOAs7KggIL/rB9hEiB3NaOPWhhoCP4vlW6MEMExGcqB/1FVxXFBPnLkEyW0Lk7ycVflZl2ocRxbfjZi0+tI1Wlinp8PvSQSc/WVrAcDgKjc/mB4ODPOyYy3G8FHgfMsrXSDEyjBKgLKMsdCrAUcqJQWjkqXleXSYOV4q3pzL+9umK+q/e3P/bIoSFQzmJKTU1eDfuvPXmow9F5H54fii/Da7ezlMJ+wPGHJrRAkmzvMbALy7xwswLhZMkOGNtRcPqaKYRmIBKpw3o6bCTtcNUHOtOQnzwY8JzrM2eBWJBXAANYw+9/ho80JIiwhg29CFNpVBuHbql2YxJQNrnl90guN65rYNpDxdIluweyUf8= anbanerj@kaermorhen" 2026-03-20 15:34:54.969483 | crc | } 2026-03-20 15:34:54.981232 | controller | changed: 2026-03-20 15:34:54.981358 | controller | { 2026-03-20 15:34:54.981410 | controller | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAMI6kkg9Wg0sG7jIJmyZemEBwUn1yzNpQQd3gnulOmZ adrianfuscoarnejo@gmail.com" 2026-03-20 15:34:54.981449 | controller | } 2026-03-20 15:34:55.206194 | controller | changed: 2026-03-20 15:34:55.206353 | controller | { 2026-03-20 15:34:55.206404 | controller | "public_key": "ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPijwpQu/3jhhhBZInXNOLEH57DrknPc3PLbsRvYyJIFzwYjX+WD4a7+nGnMYS42MuZk6TJcVqgnqofVx4isoD4= ramishra@redhat.com" 2026-03-20 15:34:55.206442 | controller | } 2026-03-20 15:34:55.328427 | crc | changed: 2026-03-20 15:34:55.328697 | crc | { 2026-03-20 15:34:55.328763 | crc | "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3VwV8Im9kRm49lt3tM36hj4Zv27FxGo4C1Q/0jqhzFmHY7RHbmeRr8ObhwWoHjXSozKWg8FL5ER0z3hTwL0W6lez3sL7hUaCmSuZmG5Hnl3x4vTSxDI9JZ/Y65rtYiiWQo2fC5xJhU/4+0e5e/pseCm8cKRSu+SaxhO+sd6FDojA2x1BzOzKiQRDy/1zWGp/cZkxcEuB1wHI5LMzN03c67vmbu+fhZRAUO4dQkvcnj2LrhQtpa+ytvnSjr8icMDosf1OsbSffwZFyHB/hfWGAfe0eIeSA2XPraxiPknXxiPKx2MJsaUTYbsZcm3EjFdHBBMumw5rBI74zLrMRvCO9GwBEmGT4rFng1nP+yw5DB8sn2zqpOsPg1LYRwCPOUveC13P6pgsZZPh812e8v5EKnETct+5XI3dVpdw6CnNiLwAyVAF15DJvBGT/u1k0Myg/bQn+Gv9k2MSj6LvQmf6WbZu2Wgjm30z3FyCneBqTL7mLF19YXzeC0ufHz5pnO1E= dasm@fedora" 2026-03-20 15:34:55.328822 | crc | } 2026-03-20 15:34:55.433041 | controller | changed: 2026-03-20 15:34:55.433298 | controller | { 2026-03-20 15:34:55.433358 | controller | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICWBreHW95Wz2Toz5YwCGQwFcUG8oFYkienDh9tntmDc ralfieri@redhat.com" 2026-03-20 15:34:55.433401 | controller | } 2026-03-20 15:34:55.651718 | controller | changed: 2026-03-20 15:34:55.651877 | controller | { 2026-03-20 15:34:55.651925 | controller | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDK0iKdi8jQTpQrDdLVH/AAgLVYyTXF7AQ1gjc/5uT3t ykarel@yatinkarel" 2026-03-20 15:34:55.651963 | controller | } 2026-03-20 15:34:55.693286 | crc | changed: 2026-03-20 15:34:55.693435 | crc | { 2026-03-20 15:34:55.693495 | crc | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHUnwjB20UKmsSed9X73eGNV5AOEFccQ3NYrRW776pEk cjeanner" 2026-03-20 15:34:55.693543 | crc | } 2026-03-20 15:34:55.860477 | controller | changed: 2026-03-20 15:34:55.860699 | controller | { 2026-03-20 15:34:55.860763 | controller | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIF/V/cLotA6LZeO32VL45Hd78skuA2lJA425Sm2LlQeZ fmount@horcrux" 2026-03-20 15:34:55.860804 | controller | } 2026-03-20 15:34:56.042637 | crc | changed: 2026-03-20 15:34:56.042836 | crc | { 2026-03-20 15:34:56.042900 | crc | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDercCMGn8rW1C4P67tHgtflPdTeXlpyUJYH+6XDd2lR jgilaber@redhat.com" 2026-03-20 15:34:56.042943 | crc | } 2026-03-20 15:34:56.105754 | controller | changed: 2026-03-20 15:34:56.106051 | controller | { 2026-03-20 15:34:56.106112 | controller | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDa7QCjuDMVmRPo1rREbGwzYeBCYVN+Ou/3WKXZEC6Sr" 2026-03-20 15:34:56.106153 | controller | } 2026-03-20 15:34:56.344196 | controller | changed: 2026-03-20 15:34:56.344356 | controller | { 2026-03-20 15:34:56.344400 | controller | "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCfNtF7NvKl915TGsGGoseUb06Hj8L/S4toWf0hExeY+F00woL6NvBlJD0nDct+P5a22I4EhvoQCRQ8reaPCm1lybR3uiRIJsj+8zkVvLwby9LXzfZorlNG9ofjd00FEmB09uW/YvTl6Q9XwwwX6tInzIOv3TMqTHHGOL74ibbj8J/FJR0cFEyj0z4WQRvtkh32xAHl83gbuINryMt0sqRI+clj2381NKL55DRLQrVw0gsfqqxiHAnXg21qWmc4J+b9e9kiuAFQjcjwTVkwJCcg3xbPwC/qokYRby/Y5S40UUd7/jEARGXT7RZgpzTuDd1oZiCVrnrqJNPaMNdVv5MLeFdf1B7iIe5aa/fGouX7AO4SdKhZUdnJmCFAGvjC6S3JMZ2wAcUl+OHnssfmdj7XL50cLo27vjuzMtLAgSqi6N99m92WCF2s8J9aVzszX7Xz9OKZCeGsiVJp3/NdABKzSEAyM9xBD/5Vho894Sav+otpySHe3p6RUTgbB5Zu8VyZRZ/UtB3ueXxyo764yrc6qWIDqrehm84Xm9g+/jpIBzGPl07NUNJpdt/6Sgf9RIKXw/7XypO5yZfUcuFNGTxLfqjTNrtgLZNcjfav6sSdVXVcMPL//XNuRdKmVFaO76eV/oGMQGr1fGcCD+N+CpI7+Q+fCNB6VFWG4nZFuI/Iuw== averdagu@redhat.com" 2026-03-20 15:34:56.344433 | controller | } 2026-03-20 15:34:56.417572 | crc | changed: 2026-03-20 15:34:56.417756 | crc | { 2026-03-20 15:34:56.417809 | crc | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAMI6kkg9Wg0sG7jIJmyZemEBwUn1yzNpQQd3gnulOmZ adrianfuscoarnejo@gmail.com" 2026-03-20 15:34:56.417849 | crc | } 2026-03-20 15:34:56.596068 | controller | changed: 2026-03-20 15:34:56.596268 | controller | { 2026-03-20 15:34:56.596336 | controller | "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDq8l27xI+QlQVdS4djp9ogSoyrNE2+Ox6vKPdhSNL1J3PE5w+WCSvMz9A5gnNuH810zwbekEApbxTze/gLQJwBHA52CChfURpXrFaxY7ePXRElwKAL3mJfzBWY/c5jnNL9TCVmFJTGZkFZP3Nh+BMgZvL6xBkt3WKm6Uq18qzd9XeKcZusrA+O+uLv1fVeQnadY9RIqOCyeFYCzLWrUfTyE8x/XG0hAWIM7qpnF2cALQS2h9n4hW5ybiUN790H08wf9hFwEf5nxY9Z9dVkPFQiTSGKNBzmnCXU9skxS/xhpFjJ5duGSZdtAHe9O+nGZm9c67hxgtf8e5PDuqAdXEv2cf6e3VBAt+Bz8EKI3yosTj0oZHfwr42Yzb1l/SKy14Rggsrc9KAQlrGXan6+u2jcQqqx7l+SWmnpFiWTV9u5cWj2IgOhApOitmRBPYqk9rE2usfO0hLn/Pj/R/Nau4803e1/EikdLE7Ps95s9mX5jRDjAoUa2JwFF5RsVFyL910= ashigupt@ashigupt.remote.csb" 2026-03-20 15:34:56.596392 | controller | } 2026-03-20 15:34:56.791320 | crc | changed: 2026-03-20 15:34:56.791527 | crc | { 2026-03-20 15:34:56.791589 | crc | "public_key": "ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPijwpQu/3jhhhBZInXNOLEH57DrknPc3PLbsRvYyJIFzwYjX+WD4a7+nGnMYS42MuZk6TJcVqgnqofVx4isoD4= ramishra@redhat.com" 2026-03-20 15:34:56.791638 | crc | } 2026-03-20 15:34:56.864392 | controller | changed: 2026-03-20 15:34:56.864724 | controller | { 2026-03-20 15:34:56.864791 | controller | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOKLl0NYKwoZ/JY5KeZU8VwRAggeOxqQJeoqp3dsAaY9" 2026-03-20 15:34:56.864834 | controller | } 2026-03-20 15:34:57.132860 | controller | changed: 2026-03-20 15:34:57.133031 | controller | { 2026-03-20 15:34:57.133084 | controller | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIASASQOH2BcOyLKuuDOdWZlPi2orcjcA8q4400T73DLH evallesp@fedora" 2026-03-20 15:34:57.133123 | controller | } 2026-03-20 15:34:57.162627 | crc | changed: 2026-03-20 15:34:57.162772 | crc | { 2026-03-20 15:34:57.162796 | crc | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICWBreHW95Wz2Toz5YwCGQwFcUG8oFYkienDh9tntmDc ralfieri@redhat.com" 2026-03-20 15:34:57.162814 | crc | } 2026-03-20 15:34:57.378479 | controller | changed: 2026-03-20 15:34:57.378642 | controller | { 2026-03-20 15:34:57.378724 | controller | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILeBWlamUph+jRKV2qrx1PGU7vWuGIt5+z9k96I8WehW amsinha@amsinha-mac" 2026-03-20 15:34:57.378772 | controller | } 2026-03-20 15:34:57.556607 | crc | changed: 2026-03-20 15:34:57.556898 | crc | { 2026-03-20 15:34:57.556976 | crc | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDK0iKdi8jQTpQrDdLVH/AAgLVYyTXF7AQ1gjc/5uT3t ykarel@yatinkarel" 2026-03-20 15:34:57.557035 | crc | } 2026-03-20 15:34:57.629861 | controller | changed: 2026-03-20 15:34:57.630004 | controller | { 2026-03-20 15:34:57.630053 | controller | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIANvVgvJBlK3gb1yz5uef/JqIGq4HLEmY2dYA8e37swb morenod@redhat-laptop" 2026-03-20 15:34:57.630091 | controller | } 2026-03-20 15:34:57.875282 | controller | changed: 2026-03-20 15:34:57.875476 | controller | { 2026-03-20 15:34:57.875555 | controller | "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZdI7t1cxYx65heVI24HTV4F7oQLW1zyfxHreL2TIJKxjyrUUKIFEUmTutcBlJRLNT2Eoix6x1sOw9YrchloCLcn//SGfTElr9mSc5jbjb7QXEU+zJMhtxyEJ1Po3CUGnj7ckiIXw7wcawZtrEOAQ9pH3ExYCJcEMiyNjRQZCxT3tPK+S4B95EWh5Fsrz9CkwpjNRPPH7LigCeQTM3Wc7r97utAslBUUvYceDSLA7rMgkitJE38b7rZBeYzsGQ8YYUBjTCtehqQXxCRjizbHWaaZkBU+N3zkKB6n/iCNGIO690NK7A/qb6msTijiz1PeuM8ThOsi9qXnbX5v0PoTpcFSojV7NHAQ71f0XXuS43FhZctT+Dcx44dT8Fb5vJu2cJGrk+qF8ZgJYNpRS7gPg0EG2EqjK7JMf9ULdjSu0r+KlqIAyLvtzT4eOnQipoKlb/WG5D/0ohKv7OMQ352ggfkBFIQsRXyyTCT98Ft9juqPuahi3CAQmP4H9dyE+7+Kz437PEtsxLmfm6naNmWi7Ee1DqWPwS8rEajsm4sNM4wW9gdBboJQtc0uZw0DfLj1I9r3Mc8Ol0jYtz0yNQDSzVLrGCaJlC311trU70tZ+ZkAVV6Mn8lOhSbj1cK0lvSr6ZK4dgqGl3I1eTZJJhbLNdg7UOVaiRx9543+C/p/As7w== brjackma@redhat.com" 2026-03-20 15:34:57.875612 | controller | } 2026-03-20 15:34:57.958768 | crc | changed: 2026-03-20 15:34:57.958938 | crc | { 2026-03-20 15:34:57.958988 | crc | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIF/V/cLotA6LZeO32VL45Hd78skuA2lJA425Sm2LlQeZ fmount@horcrux" 2026-03-20 15:34:57.959029 | crc | } 2026-03-20 15:34:58.105039 | controller | changed: 2026-03-20 15:34:58.105194 | controller | { 2026-03-20 15:34:58.105246 | controller | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKwedoZ0TWPJX/z/4TAbO/kKcDZOQVgRH0hAqrL5UCI1 vcastell@redhat.com" 2026-03-20 15:34:58.105289 | controller | } 2026-03-20 15:34:58.350075 | crc | changed: 2026-03-20 15:34:58.350191 | crc | { 2026-03-20 15:34:58.350213 | crc | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDa7QCjuDMVmRPo1rREbGwzYeBCYVN+Ou/3WKXZEC6Sr" 2026-03-20 15:34:58.350231 | crc | } 2026-03-20 15:34:58.367422 | controller | changed: 2026-03-20 15:34:58.367469 | controller | { 2026-03-20 15:34:58.367491 | controller | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEmv8sE8GCk6ZTPIqF0FQrttBdL3mq7rCm/IJy0xDFh7 michburk@redhat.com" 2026-03-20 15:34:58.367509 | controller | } 2026-03-20 15:34:58.622641 | controller | changed: 2026-03-20 15:34:58.622855 | controller | { 2026-03-20 15:34:58.622905 | controller | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICy6GpGEtwevXEEn4mmLR5lmSLe23dGgAvzkB9DMNbkf rsafrono@rsafrono" 2026-03-20 15:34:58.622943 | controller | } 2026-03-20 15:34:58.718800 | crc | changed: 2026-03-20 15:34:58.719083 | crc | { 2026-03-20 15:34:58.719155 | crc | "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCfNtF7NvKl915TGsGGoseUb06Hj8L/S4toWf0hExeY+F00woL6NvBlJD0nDct+P5a22I4EhvoQCRQ8reaPCm1lybR3uiRIJsj+8zkVvLwby9LXzfZorlNG9ofjd00FEmB09uW/YvTl6Q9XwwwX6tInzIOv3TMqTHHGOL74ibbj8J/FJR0cFEyj0z4WQRvtkh32xAHl83gbuINryMt0sqRI+clj2381NKL55DRLQrVw0gsfqqxiHAnXg21qWmc4J+b9e9kiuAFQjcjwTVkwJCcg3xbPwC/qokYRby/Y5S40UUd7/jEARGXT7RZgpzTuDd1oZiCVrnrqJNPaMNdVv5MLeFdf1B7iIe5aa/fGouX7AO4SdKhZUdnJmCFAGvjC6S3JMZ2wAcUl+OHnssfmdj7XL50cLo27vjuzMtLAgSqi6N99m92WCF2s8J9aVzszX7Xz9OKZCeGsiVJp3/NdABKzSEAyM9xBD/5Vho894Sav+otpySHe3p6RUTgbB5Zu8VyZRZ/UtB3ueXxyo764yrc6qWIDqrehm84Xm9g+/jpIBzGPl07NUNJpdt/6Sgf9RIKXw/7XypO5yZfUcuFNGTxLfqjTNrtgLZNcjfav6sSdVXVcMPL//XNuRdKmVFaO76eV/oGMQGr1fGcCD+N+CpI7+Q+fCNB6VFWG4nZFuI/Iuw== averdagu@redhat.com" 2026-03-20 15:34:58.719199 | crc | } 2026-03-20 15:34:58.719259 | crc | changed: All items complete 2026-03-20 15:34:58.719301 | 2026-03-20 15:34:59.073499 | crc | changed: 2026-03-20 15:34:59.073620 | crc | { 2026-03-20 15:34:59.073643 | crc | "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDq8l27xI+QlQVdS4djp9ogSoyrNE2+Ox6vKPdhSNL1J3PE5w+WCSvMz9A5gnNuH810zwbekEApbxTze/gLQJwBHA52CChfURpXrFaxY7ePXRElwKAL3mJfzBWY/c5jnNL9TCVmFJTGZkFZP3Nh+BMgZvL6xBkt3WKm6Uq18qzd9XeKcZusrA+O+uLv1fVeQnadY9RIqOCyeFYCzLWrUfTyE8x/XG0hAWIM7qpnF2cALQS2h9n4hW5ybiUN790H08wf9hFwEf5nxY9Z9dVkPFQiTSGKNBzmnCXU9skxS/xhpFjJ5duGSZdtAHe9O+nGZm9c67hxgtf8e5PDuqAdXEv2cf6e3VBAt+Bz8EKI3yosTj0oZHfwr42Yzb1l/SKy14Rggsrc9KAQlrGXan6+u2jcQqqx7l+SWmnpFiWTV9u5cWj2IgOhApOitmRBPYqk9rE2usfO0hLn/Pj/R/Nau4803e1/EikdLE7Ps95s9mX5jRDjAoUa2JwFF5RsVFyL910= ashigupt@ashigupt.remote.csb" 2026-03-20 15:34:59.073661 | crc | } 2026-03-20 15:34:59.465902 | crc | changed: 2026-03-20 15:34:59.466062 | crc | { 2026-03-20 15:34:59.466116 | crc | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOKLl0NYKwoZ/JY5KeZU8VwRAggeOxqQJeoqp3dsAaY9" 2026-03-20 15:34:59.466184 | crc | } 2026-03-20 15:34:59.813946 | crc | changed: 2026-03-20 15:34:59.814121 | crc | { 2026-03-20 15:34:59.814192 | crc | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIASASQOH2BcOyLKuuDOdWZlPi2orcjcA8q4400T73DLH evallesp@fedora" 2026-03-20 15:34:59.814253 | crc | } 2026-03-20 15:35:00.179634 | crc | changed: 2026-03-20 15:35:00.179823 | crc | { 2026-03-20 15:35:00.179867 | crc | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILeBWlamUph+jRKV2qrx1PGU7vWuGIt5+z9k96I8WehW amsinha@amsinha-mac" 2026-03-20 15:35:00.179927 | crc | } 2026-03-20 15:35:00.549573 | crc | changed: 2026-03-20 15:35:00.549792 | crc | { 2026-03-20 15:35:00.549862 | crc | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIANvVgvJBlK3gb1yz5uef/JqIGq4HLEmY2dYA8e37swb morenod@redhat-laptop" 2026-03-20 15:35:00.549917 | crc | } 2026-03-20 15:35:00.931325 | crc | changed: 2026-03-20 15:35:00.931520 | crc | { 2026-03-20 15:35:00.931597 | crc | "public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZdI7t1cxYx65heVI24HTV4F7oQLW1zyfxHreL2TIJKxjyrUUKIFEUmTutcBlJRLNT2Eoix6x1sOw9YrchloCLcn//SGfTElr9mSc5jbjb7QXEU+zJMhtxyEJ1Po3CUGnj7ckiIXw7wcawZtrEOAQ9pH3ExYCJcEMiyNjRQZCxT3tPK+S4B95EWh5Fsrz9CkwpjNRPPH7LigCeQTM3Wc7r97utAslBUUvYceDSLA7rMgkitJE38b7rZBeYzsGQ8YYUBjTCtehqQXxCRjizbHWaaZkBU+N3zkKB6n/iCNGIO690NK7A/qb6msTijiz1PeuM8ThOsi9qXnbX5v0PoTpcFSojV7NHAQ71f0XXuS43FhZctT+Dcx44dT8Fb5vJu2cJGrk+qF8ZgJYNpRS7gPg0EG2EqjK7JMf9ULdjSu0r+KlqIAyLvtzT4eOnQipoKlb/WG5D/0ohKv7OMQ352ggfkBFIQsRXyyTCT98Ft9juqPuahi3CAQmP4H9dyE+7+Kz437PEtsxLmfm6naNmWi7Ee1DqWPwS8rEajsm4sNM4wW9gdBboJQtc0uZw0DfLj1I9r3Mc8Ol0jYtz0yNQDSzVLrGCaJlC311trU70tZ+ZkAVV6Mn8lOhSbj1cK0lvSr6ZK4dgqGl3I1eTZJJhbLNdg7UOVaiRx9543+C/p/As7w== brjackma@redhat.com" 2026-03-20 15:35:00.931643 | crc | } 2026-03-20 15:35:01.283875 | crc | changed: 2026-03-20 15:35:01.284153 | crc | { 2026-03-20 15:35:01.284244 | crc | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKwedoZ0TWPJX/z/4TAbO/kKcDZOQVgRH0hAqrL5UCI1 vcastell@redhat.com" 2026-03-20 15:35:01.284319 | crc | } 2026-03-20 15:35:01.619423 | crc | changed: 2026-03-20 15:35:01.619708 | crc | { 2026-03-20 15:35:01.619777 | crc | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEmv8sE8GCk6ZTPIqF0FQrttBdL3mq7rCm/IJy0xDFh7 michburk@redhat.com" 2026-03-20 15:35:01.619834 | crc | } 2026-03-20 15:35:01.998364 | crc | changed: 2026-03-20 15:35:01.998715 | crc | { 2026-03-20 15:35:01.998785 | crc | "public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICy6GpGEtwevXEEn4mmLR5lmSLe23dGgAvzkB9DMNbkf rsafrono@rsafrono" 2026-03-20 15:35:01.998827 | crc | } 2026-03-20 15:35:02.060269 | 2026-03-20 15:35:02.060411 | TASK [Set timezone to UTC] 2026-03-20 15:35:02.600810 | controller | changed: executed `/bin/timedatectl set-timezone UTC` 2026-03-20 15:35:03.714209 | crc | ok 2026-03-20 15:35:03.726601 | 2026-03-20 15:35:03.726789 | TASK [Create nodepool directory] 2026-03-20 15:35:04.014083 | controller | changed 2026-03-20 15:35:04.069972 | crc | changed 2026-03-20 15:35:04.082529 | 2026-03-20 15:35:04.082696 | TASK [Create nodepool sub_nodes file] 2026-03-20 15:35:04.736256 | controller | changed 2026-03-20 15:35:04.987718 | crc | changed 2026-03-20 15:35:04.998605 | 2026-03-20 15:35:04.998779 | TASK [Create nodepool sub_nodes_private file] 2026-03-20 15:35:05.674895 | controller | changed 2026-03-20 15:35:05.925408 | crc | changed 2026-03-20 15:35:05.936536 | 2026-03-20 15:35:05.936659 | LOOP [Populate nodepool sub_nodes file] 2026-03-20 15:35:06.010849 | 2026-03-20 15:35:06.011149 | LOOP [Populate nodepool sub_nodes_private file] 2026-03-20 15:35:06.078754 | 2026-03-20 15:35:06.078959 | TASK [Create nodepool primary file] 2026-03-20 15:35:06.112491 | controller | skipping: Conditional result was False 2026-03-20 15:35:06.134190 | crc | skipping: Conditional result was False 2026-03-20 15:35:06.143974 | 2026-03-20 15:35:06.144060 | TASK [Create nodepool node_private for this node] 2026-03-20 15:35:06.785428 | controller | changed 2026-03-20 15:35:07.076478 | crc | changed 2026-03-20 15:35:07.124690 | 2026-03-20 15:35:07.124828 | LOOP [Copy ssh keys to nodepool directory] 2026-03-20 15:35:07.525155 | controller | ok: Item: id_rsa Runtime: 0:00:00.003515 2026-03-20 15:35:07.525518 | 2026-03-20 15:35:07.566813 | crc | ok: Item: id_rsa Runtime: 0:00:00.006557 2026-03-20 15:35:07.706142 | controller | ok: Item: id_rsa.pub Runtime: 0:00:00.003531 2026-03-20 15:35:07.843506 | crc | ok: Item: id_rsa.pub Runtime: 0:00:00.005737 2026-03-20 15:35:07.855926 | 2026-03-20 15:35:07.856069 | TASK [Add sudoers role for zuul-sudo-grep.sh] 2026-03-20 15:35:08.514344 | controller | changed 2026-03-20 15:35:08.754682 | crc | changed 2026-03-20 15:35:08.759574 | 2026-03-20 15:35:08.759633 | TASK [Validate sudoers config after edits] 2026-03-20 15:35:09.091461 | controller | /etc/sudoers: parsed OK 2026-03-20 15:35:09.091535 | controller | /etc/sudoers.d/90-cloud-init-users: parsed OK 2026-03-20 15:35:09.091593 | controller | /etc/sudoers.d/zuul: parsed OK 2026-03-20 15:35:09.091599 | controller | /etc/sudoers.d/zuul-sudo-grep: parsed OK 2026-03-20 15:35:09.107316 | crc | /etc/sudoers: parsed OK 2026-03-20 15:35:09.107530 | crc | /etc/sudoers.d/coreos-sudo-group: parsed OK 2026-03-20 15:35:09.107548 | crc | /etc/sudoers.d/zuul-sudo-grep: parsed OK 2026-03-20 15:35:09.317615 | controller | ok: Runtime: 0:00:00.006757 2026-03-20 15:35:09.332587 | crc | ok: Runtime: 0:00:00.009598 2026-03-20 15:35:09.343434 | 2026-03-20 15:35:09.343560 | TASK [Show the environment passed in to job shell scripts] 2026-03-20 15:35:09.673821 | controller | SHELL=/bin/bash 2026-03-20 15:35:09.673856 | controller | ZUUL_PROJECT=openstack-k8s-operators/watcher-operator 2026-03-20 15:35:09.673864 | controller | ZUUL_SHORT_PROJECT_NAME=watcher-operator 2026-03-20 15:35:09.673869 | controller | ZUUL_CHANGES=openstack-k8s-operators/watcher-operator:main:refs/changes/54/354/ee00c2d330b27d46c48ac29a20680b56ca50df3c 2026-03-20 15:35:09.673874 | controller | PWD=/home/zuul 2026-03-20 15:35:09.673879 | controller | ZUUL_PIPELINE=github-check 2026-03-20 15:35:09.673884 | controller | LOGNAME=zuul 2026-03-20 15:35:09.673888 | controller | XDG_SESSION_TYPE=tty 2026-03-20 15:35:09.673893 | controller | _=/usr/bin/env 2026-03-20 15:35:09.673898 | controller | MOTD_SHOWN=pam 2026-03-20 15:35:09.673902 | controller | HOME=/home/zuul 2026-03-20 15:35:09.673907 | controller | LANG=en_US.UTF-8 2026-03-20 15:35:09.673911 | controller | SSH_CONNECTION=38.102.83.114 41322 38.102.83.159 22 2026-03-20 15:35:09.673915 | controller | TOX_TESTENV_PASSENV=ZUUL_UUID ZUUL_PROJECT ZUUL_SHORT_PROJECT_NAME ZUUL_PIPELINE ZUUL_VOTING WORKSPACE BUILD_TIMEOUT ZUUL_BRANCH ZUUL_CHANGES ZUUL_CHANGE_IDS ZUUL_CHANGE ZUUL_PATCHSET 2026-03-20 15:35:09.673922 | controller | ZUUL_CHANGE_IDS=354,ee00c2d330b27d46c48ac29a20680b56ca50df3c 2026-03-20 15:35:09.673926 | controller | WORKSPACE=/home/zuul/workspace 2026-03-20 15:35:09.673931 | controller | XDG_SESSION_CLASS=user 2026-03-20 15:35:09.673935 | controller | SELINUX_ROLE_REQUESTED= 2026-03-20 15:35:09.673940 | controller | LESSOPEN=||/usr/bin/lesspipe.sh %s 2026-03-20 15:35:09.673945 | controller | USER=zuul 2026-03-20 15:35:09.673949 | controller | ZUUL_VOTING=True 2026-03-20 15:35:09.673954 | controller | BUILD_TIMEOUT=7200000 2026-03-20 15:35:09.673958 | controller | SELINUX_USE_CURRENT_RANGE= 2026-03-20 15:35:09.673962 | controller | SHLVL=1 2026-03-20 15:35:09.673967 | controller | ZUUL_PATCHSET=ee00c2d330b27d46c48ac29a20680b56ca50df3c 2026-03-20 15:35:09.673971 | controller | XDG_SESSION_ID=1 2026-03-20 15:35:09.673976 | controller | ZUUL_BRANCH=main 2026-03-20 15:35:09.673980 | controller | XDG_RUNTIME_DIR=/run/user/1000 2026-03-20 15:35:09.673984 | controller | SSH_CLIENT=38.102.83.114 41322 22 2026-03-20 15:35:09.673989 | controller | DEBUGINFOD_URLS=https://debuginfod.centos.org/ 2026-03-20 15:35:09.673993 | controller | DEBUGINFOD_IMA_CERT_PATH=/etc/keys/ima: 2026-03-20 15:35:09.673998 | controller | which_declare=declare -f 2026-03-20 15:35:09.674002 | controller | PATH=/home/zuul/.local/bin:/home/zuul/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin 2026-03-20 15:35:09.674007 | controller | SELINUX_LEVEL_REQUESTED= 2026-03-20 15:35:09.674011 | controller | ZUUL_CHANGE=354 2026-03-20 15:35:09.674016 | controller | DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus 2026-03-20 15:35:09.674020 | controller | ZUUL_UUID=7ffee4ddf4ea401db94c9ba86e43148b 2026-03-20 15:35:09.674024 | controller | BASH_FUNC_which%%=() { ( alias; 2026-03-20 15:35:09.674029 | controller | eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot $@ 2026-03-20 15:35:09.674033 | controller | } 2026-03-20 15:35:09.726027 | crc | SHELL=/bin/bash 2026-03-20 15:35:09.726118 | crc | ZUUL_PROJECT=openstack-k8s-operators/watcher-operator 2026-03-20 15:35:09.726143 | crc | ZUUL_SHORT_PROJECT_NAME=watcher-operator 2026-03-20 15:35:09.726161 | crc | ZUUL_CHANGES=openstack-k8s-operators/watcher-operator:main:refs/changes/54/354/ee00c2d330b27d46c48ac29a20680b56ca50df3c 2026-03-20 15:35:09.726179 | crc | PWD=/var/home/core 2026-03-20 15:35:09.726193 | crc | ZUUL_PIPELINE=github-check 2026-03-20 15:35:09.726205 | crc | LOGNAME=core 2026-03-20 15:35:09.726218 | crc | XDG_SESSION_TYPE=tty 2026-03-20 15:35:09.726230 | crc | _=/usr/bin/env 2026-03-20 15:35:09.726243 | crc | MOTD_SHOWN=pam 2026-03-20 15:35:09.726292 | crc | HOME=/var/home/core 2026-03-20 15:35:09.726305 | crc | LANG=C.UTF-8 2026-03-20 15:35:09.726318 | crc | SSH_CONNECTION=38.102.83.114 55326 38.102.83.181 22 2026-03-20 15:35:09.726331 | crc | TOX_TESTENV_PASSENV=ZUUL_UUID ZUUL_PROJECT ZUUL_SHORT_PROJECT_NAME ZUUL_PIPELINE ZUUL_VOTING WORKSPACE BUILD_TIMEOUT ZUUL_BRANCH ZUUL_CHANGES ZUUL_CHANGE_IDS ZUUL_CHANGE ZUUL_PATCHSET 2026-03-20 15:35:09.726347 | crc | ZUUL_CHANGE_IDS=354,ee00c2d330b27d46c48ac29a20680b56ca50df3c 2026-03-20 15:35:09.726369 | crc | WORKSPACE=/home/zuul/workspace 2026-03-20 15:35:09.726382 | crc | XDG_SESSION_CLASS=user 2026-03-20 15:35:09.726394 | crc | SELINUX_ROLE_REQUESTED= 2026-03-20 15:35:09.726407 | crc | LESSOPEN=||/usr/bin/lesspipe.sh %s 2026-03-20 15:35:09.726419 | crc | USER=core 2026-03-20 15:35:09.726432 | crc | ZUUL_VOTING=True 2026-03-20 15:35:09.726444 | crc | BUILD_TIMEOUT=7200000 2026-03-20 15:35:09.726456 | crc | SELINUX_USE_CURRENT_RANGE= 2026-03-20 15:35:09.726468 | crc | SHLVL=1 2026-03-20 15:35:09.726481 | crc | ZUUL_PATCHSET=ee00c2d330b27d46c48ac29a20680b56ca50df3c 2026-03-20 15:35:09.726493 | crc | XDG_SESSION_ID=2 2026-03-20 15:35:09.726506 | crc | ZUUL_BRANCH=main 2026-03-20 15:35:09.726518 | crc | XDG_RUNTIME_DIR=/run/user/1000 2026-03-20 15:35:09.726530 | crc | SSH_CLIENT=38.102.83.114 55326 22 2026-03-20 15:35:09.726546 | crc | which_declare=declare -f 2026-03-20 15:35:09.726558 | crc | PATH=/var/home/core/.local/bin:/var/home/core/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin 2026-03-20 15:35:09.726571 | crc | SELINUX_LEVEL_REQUESTED= 2026-03-20 15:35:09.726583 | crc | ZUUL_CHANGE=354 2026-03-20 15:35:09.726596 | crc | DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus 2026-03-20 15:35:09.726608 | crc | ZUUL_UUID=7ffee4ddf4ea401db94c9ba86e43148b 2026-03-20 15:35:09.726621 | crc | BASH_FUNC_which%%=() { ( alias; 2026-03-20 15:35:09.726633 | crc | eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot $@ 2026-03-20 15:35:09.726646 | crc | } 2026-03-20 15:35:09.934795 | controller | ok: Runtime: 0:00:00.005138 2026-03-20 15:35:09.949605 | crc | ok: Runtime: 0:00:00.010913 2026-03-20 15:35:09.960818 | 2026-03-20 15:35:09.960940 | TASK [Workaround hardcoded /home/zuul/workspace in zuul_legacy_vars] 2026-03-20 15:35:09.995290 | controller | skipping: Conditional result was False 2026-03-20 15:35:10.006795 | crc | skipping: Conditional result was False 2026-03-20 15:35:10.020248 | 2026-03-20 15:35:10.020408 | TASK [Symlink /home/zuul-worker/workspace] 2026-03-20 15:35:10.598792 | controller | skipping: Conditional result was False 2026-03-20 15:35:10.599539 | crc | skipping: Conditional result was False 2026-03-20 15:35:10.611162 | 2026-03-20 15:35:10.611292 | TASK [Ensure legacy workspace directory] 2026-03-20 15:35:10.896420 | controller | changed 2026-03-20 15:35:10.949491 | crc | changed 2026-03-20 15:35:10.996346 | 2026-03-20 15:35:10.996401 | PLAY RECAP 2026-03-20 15:35:10.996441 | controller | ok: 14 changed: 11 unreachable: 0 failed: 0 skipped: 5 rescued: 0 ignored: 0 2026-03-20 15:35:10.996466 | crc | ok: 14 changed: 10 unreachable: 0 failed: 0 skipped: 5 rescued: 0 ignored: 0 2026-03-20 15:35:10.996482 | 2026-03-20 15:35:11.094837 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/ci-framework-rdo-base/pre.yaml@master] 2026-03-20 15:35:11.105312 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/crc/bootstrap-ci-network.yml@master] 2026-03-20 15:35:11.688704 | 2026-03-20 15:35:11.688808 | PLAY [controller] 2026-03-20 15:35:11.710528 | 2026-03-20 15:35:11.710604 | TASK [Debug parameters] 2026-03-20 15:35:11.749551 | controller | ok 2026-03-20 15:35:11.754873 | 2026-03-20 15:35:11.754940 | TASK [Assert crc_ci_bootstrap_cloud_name is defined] 2026-03-20 15:35:11.805317 | controller | ok: All assertions passed 2026-03-20 15:35:11.809899 | 2026-03-20 15:35:11.809961 | TASK [Assert crc_ci_bootstrap_networking is defined and contains the minimum fields] 2026-03-20 15:35:11.858485 | controller | ok: All assertions passed 2026-03-20 15:35:11.863004 | 2026-03-20 15:35:11.863064 | TASK [Assert that the default network exists and has no VLAN tag] 2026-03-20 15:35:11.892338 | controller | ok: All assertions passed 2026-03-20 15:35:11.896728 | 2026-03-20 15:35:11.896785 | LOOP [Assert that all instances has a given fixed IP for each net and default net is present] 2026-03-20 15:35:11.998791 | controller | ok: All assertions passed 2026-03-20 15:35:12.011204 | controller | ok: All assertions passed 2026-03-20 15:35:12.017416 | 2026-03-20 15:35:12.017497 | LOOP [Assert that every host references an valid/declared network] 2026-03-20 15:35:12.073477 | controller | ok: All assertions passed 2026-03-20 15:35:12.073987 | controller | ok: All items complete 2026-03-20 15:35:12.074057 | 2026-03-20 15:35:12.084110 | controller | ok: All assertions passed 2026-03-20 15:35:12.099905 | 2026-03-20 15:35:12.099987 | LOOP [Assert that all instances has in range fixed IP for each net] 2026-03-20 15:35:12.185459 | controller | ok: All assertions passed 2026-03-20 15:35:12.202656 | controller | ok: All assertions passed 2026-03-20 15:35:12.211771 | 2026-03-20 15:35:12.211848 | TASK [Assert that all non-default nets has the vlan tag] 2026-03-20 15:35:12.270593 | controller | ok: All assertions passed 2026-03-20 15:35:12.275123 | 2026-03-20 15:35:12.275180 | TASK [Install required packages] 2026-03-20 15:35:26.568516 | controller | changed 2026-03-20 15:35:26.580992 | 2026-03-20 15:35:26.581116 | TASK [Create openstack config dir] 2026-03-20 15:35:26.923150 | controller | changed 2026-03-20 15:35:26.934322 | 2026-03-20 15:35:26.934456 | LOOP [Create ci dir in etc folder in the controller] 2026-03-20 15:35:27.286184 | controller -> 38.102.83.181 | changed: "crc" 2026-03-20 15:35:27.286585 | controller -> 38.102.83.181 | ok: All items complete 2026-03-20 15:35:27.286643 | 2026-03-20 15:35:27.557687 | controller -> 38.102.83.159 | changed: "controller" 2026-03-20 15:35:27.575432 | 2026-03-20 15:35:27.575585 | TASK [Set global resource names as facts] 2026-03-20 15:35:27.624201 | controller | ok 2026-03-20 15:35:27.635184 | 2026-03-20 15:35:27.635312 | TASK [Generate CA cert file] 2026-03-20 15:35:27.671486 | controller | skipping: Conditional result was False 2026-03-20 15:35:27.685446 | 2026-03-20 15:35:27.685582 | TASK [Update CA certs] 2026-03-20 15:35:28.232190 | controller | skipping: Conditional result was False 2026-03-20 15:35:28.238120 | 2026-03-20 15:35:28.238182 | TASK [Generate clouds config from cloud_secrets secret] 2026-03-20 15:35:29.200817 | controller | changed 2026-03-20 15:35:29.213017 | 2026-03-20 15:35:29.213207 | TASK [Create the default network] 2026-03-20 15:35:33.072763 | controller | admin_state_up: true 2026-03-20 15:35:33.073149 | controller | availability_zone_hints: 2026-03-20 15:35:33.073452 | controller | - nova 2026-03-20 15:35:33.073470 | controller | availability_zones: [] 2026-03-20 15:35:33.073989 | controller | created_at: '2026-03-20T15:35:32Z' 2026-03-20 15:35:33.074293 | controller | description: '' 2026-03-20 15:35:33.074565 | controller | dns_domain: '' 2026-03-20 15:35:33.074927 | controller | id: adbac40f-789f-4953-a393-3ccea9f4523a 2026-03-20 15:35:33.075343 | controller | ipv4_address_scope: null 2026-03-20 15:35:33.075647 | controller | ipv6_address_scope: null 2026-03-20 15:35:33.076065 | controller | is_default: false 2026-03-20 15:35:33.076354 | controller | is_vlan_qinq: null 2026-03-20 15:35:33.076655 | controller | is_vlan_transparent: true 2026-03-20 15:35:33.076970 | controller | l2_adjacency: true 2026-03-20 15:35:33.077330 | controller | mtu: 1500 2026-03-20 15:35:33.077637 | controller | name: zuul-ci-net-7ffee4dd 2026-03-20 15:35:33.077980 | controller | port_security_enabled: false 2026-03-20 15:35:33.078353 | controller | project_id: 4b633c451ac74233be3721a3635275e5 2026-03-20 15:35:33.078658 | controller | provider:network_type: null 2026-03-20 15:35:33.078998 | controller | provider:physical_network: null 2026-03-20 15:35:33.079283 | controller | provider:segmentation_id: null 2026-03-20 15:35:33.079467 | controller | qos_policy_id: null 2026-03-20 15:35:33.079643 | controller | revision_number: 1 2026-03-20 15:35:33.079847 | controller | router:external: false 2026-03-20 15:35:33.080017 | controller | segments: null 2026-03-20 15:35:33.080183 | controller | shared: false 2026-03-20 15:35:33.080351 | controller | status: ACTIVE 2026-03-20 15:35:33.080510 | controller | subnets: [] 2026-03-20 15:35:33.080668 | controller | tags: [] 2026-03-20 15:35:33.080899 | controller | updated_at: '2026-03-20T15:35:32Z' 2026-03-20 15:35:33.267000 | controller | changed 2026-03-20 15:35:33.278329 | 2026-03-20 15:35:33.278455 | TASK [Create subnet - ipv4] 2026-03-20 15:35:37.336676 | controller | allocation_pools: 2026-03-20 15:35:37.336848 | controller | - end: 192.168.122.254 2026-03-20 15:35:37.336857 | controller | start: 192.168.122.2 2026-03-20 15:35:37.336864 | controller | cidr: 192.168.122.0/24 2026-03-20 15:35:37.336987 | controller | created_at: '2026-03-20T15:35:37Z' 2026-03-20 15:35:37.337109 | controller | description: '' 2026-03-20 15:35:37.337278 | controller | dns_nameservers: [] 2026-03-20 15:35:37.337416 | controller | dns_publish_fixed_ip: null 2026-03-20 15:35:37.337535 | controller | enable_dhcp: false 2026-03-20 15:35:37.337655 | controller | gateway_ip: 192.168.122.1 2026-03-20 15:35:37.337769 | controller | host_routes: [] 2026-03-20 15:35:37.337880 | controller | id: fbbf2376-14dd-4853-abbb-f430e1538dd0 2026-03-20 15:35:37.337997 | controller | ip_version: 4 2026-03-20 15:35:37.338093 | controller | ipv6_address_mode: null 2026-03-20 15:35:37.338185 | controller | ipv6_ra_mode: null 2026-03-20 15:35:37.338284 | controller | name: zuul-ci-subnet-7ffee4dd 2026-03-20 15:35:37.338398 | controller | network_id: adbac40f-789f-4953-a393-3ccea9f4523a 2026-03-20 15:35:37.338510 | controller | project_id: 4b633c451ac74233be3721a3635275e5 2026-03-20 15:35:37.338605 | controller | revision_number: 0 2026-03-20 15:35:37.338711 | controller | segment_id: null 2026-03-20 15:35:37.338810 | controller | service_types: [] 2026-03-20 15:35:37.338906 | controller | subnetpool_id: null 2026-03-20 15:35:37.338993 | controller | tags: [] 2026-03-20 15:35:37.339106 | controller | updated_at: '2026-03-20T15:35:37Z' 2026-03-20 15:35:37.833291 | controller | changed 2026-03-20 15:35:37.845019 | 2026-03-20 15:35:37.845158 | TASK [Set yaml returned data for further usage] 2026-03-20 15:35:37.924960 | controller | ok 2026-03-20 15:35:37.936819 | 2026-03-20 15:35:37.936948 | TASK [Create router] 2026-03-20 15:35:38.486232 | controller | skipping: Conditional result was False 2026-03-20 15:35:38.494521 | 2026-03-20 15:35:38.494610 | TASK [Set router yaml returned data for further usage] 2026-03-20 15:35:38.539952 | controller | skipping: Conditional result was False 2026-03-20 15:35:38.547822 | 2026-03-20 15:35:38.547915 | TASK [Add router to subnet] 2026-03-20 15:35:39.088104 | controller | skipping: Conditional result was False 2026-03-20 15:35:39.096765 | 2026-03-20 15:35:39.096856 | LOOP [Create ports, VLANs and configure them on the instance] 2026-03-20 15:35:39.136747 | controller | ok: 2026-03-20 15:35:39.136956 | controller | { 2026-03-20 15:35:39.136982 | controller | "key": "controller", 2026-03-20 15:35:39.136999 | controller | "value": { 2026-03-20 15:35:39.137015 | controller | "networks": { 2026-03-20 15:35:39.137030 | controller | "default": { 2026-03-20 15:35:39.137045 | controller | "ip": "192.168.122.11" 2026-03-20 15:35:39.137060 | controller | } 2026-03-20 15:35:39.137074 | controller | } 2026-03-20 15:35:39.137088 | controller | } 2026-03-20 15:35:39.137102 | controller | } 2026-03-20 15:35:42.143962 | controller | ok: 2026-03-20 15:35:42.144110 | controller | { 2026-03-20 15:35:42.144160 | controller | "key": "crc", 2026-03-20 15:35:42.144199 | controller | "value": { 2026-03-20 15:35:42.144235 | controller | "networks": { 2026-03-20 15:35:42.144271 | controller | "default": { 2026-03-20 15:35:42.144307 | controller | "ip": "192.168.122.10" 2026-03-20 15:35:42.144343 | controller | }, 2026-03-20 15:35:42.144377 | controller | "internal-api": { 2026-03-20 15:35:42.144412 | controller | "ip": "172.17.0.5" 2026-03-20 15:35:42.144446 | controller | }, 2026-03-20 15:35:42.144480 | controller | "storage": { 2026-03-20 15:35:42.144513 | controller | "ip": "172.18.0.5" 2026-03-20 15:35:42.144546 | controller | }, 2026-03-20 15:35:42.144579 | controller | "tenant": { 2026-03-20 15:35:42.144611 | controller | "ip": "172.19.0.5" 2026-03-20 15:35:42.144644 | controller | } 2026-03-20 15:35:42.144720 | controller | } 2026-03-20 15:35:42.144758 | controller | } 2026-03-20 15:35:42.144792 | controller | } 2026-03-20 15:35:42.166640 | controller | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/trusted/project_0/review.rdoproject.org/config/playbooks/crc/bootstrap-ci-network-instance.yml 2026-03-20 15:35:42.174768 | controller | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/trusted/project_0/review.rdoproject.org/config/playbooks/crc/bootstrap-ci-network-instance.yml 2026-03-20 15:35:42.195788 | 2026-03-20 15:35:42.195931 | TASK [Create instance ports and attach if necessary] 2026-03-20 15:35:42.220338 | controller | ok 2026-03-20 15:35:42.298738 | controller | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/trusted/project_0/review.rdoproject.org/config/playbooks/crc/bootstrap-ci-network-instance-create-attach-port.yml 2026-03-20 15:35:42.309006 | 2026-03-20 15:35:42.309087 | TASK [Print the host networking data] 2026-03-20 15:35:42.351192 | controller | ok 2026-03-20 15:35:42.358175 | 2026-03-20 15:35:42.358256 | TASK [Create the private network parent port] 2026-03-20 15:35:45.033639 | controller | admin_state_up: true 2026-03-20 15:35:45.033724 | controller | allowed_address_pairs: [] 2026-03-20 15:35:45.033897 | controller | binding_host_id: null 2026-03-20 15:35:45.034005 | controller | binding_profile: {} 2026-03-20 15:35:45.034126 | controller | binding_vif_details: {} 2026-03-20 15:35:45.034228 | controller | binding_vif_type: null 2026-03-20 15:35:45.034363 | controller | binding_vnic_type: normal 2026-03-20 15:35:45.034480 | controller | created_at: '2026-03-20T15:35:44Z' 2026-03-20 15:35:45.034586 | controller | data_plane_status: null 2026-03-20 15:35:45.034679 | controller | description: '' 2026-03-20 15:35:45.034783 | controller | device_id: '' 2026-03-20 15:35:45.034874 | controller | device_owner: '' 2026-03-20 15:35:45.034970 | controller | device_profile: null 2026-03-20 15:35:45.035264 | controller | dns_assignment: 2026-03-20 15:35:45.035358 | controller | - fqdn: host-192-168-122-11.openstacklocal. 2026-03-20 15:35:45.035366 | controller | hostname: host-192-168-122-11 2026-03-20 15:35:45.035371 | controller | ip_address: 192.168.122.11 2026-03-20 15:35:45.035377 | controller | dns_domain: '' 2026-03-20 15:35:45.035447 | controller | dns_name: '' 2026-03-20 15:35:45.035552 | controller | extra_dhcp_opts: [] 2026-03-20 15:35:45.035801 | controller | fixed_ips: 2026-03-20 15:35:45.035906 | controller | - ip_address: 192.168.122.11 2026-03-20 15:35:45.035913 | controller | subnet_id: fbbf2376-14dd-4853-abbb-f430e1538dd0 2026-03-20 15:35:45.035919 | controller | hardware_offload_type: null 2026-03-20 15:35:45.035992 | controller | hints: '' 2026-03-20 15:35:45.036103 | controller | id: 30e4281f-5f88-42bc-a0da-d9e23bfff189 2026-03-20 15:35:45.036202 | controller | ip_allocation: immediate 2026-03-20 15:35:45.036307 | controller | mac_address: fa:16:3e:0d:21:04 2026-03-20 15:35:45.036427 | controller | name: controller-ca1c710b-d9a5-466b-a8d0-123e17aad387 2026-03-20 15:35:45.036551 | controller | network_id: adbac40f-789f-4953-a393-3ccea9f4523a 2026-03-20 15:35:45.036654 | controller | numa_affinity_policy: null 2026-03-20 15:35:45.036809 | controller | port_security_enabled: false 2026-03-20 15:35:45.036924 | controller | project_id: 4b633c451ac74233be3721a3635275e5 2026-03-20 15:35:45.037022 | controller | propagate_uplink_status: null 2026-03-20 15:35:45.037119 | controller | resource_request: null 2026-03-20 15:35:45.037239 | controller | revision_number: 1 2026-03-20 15:35:45.037339 | controller | qos_network_policy_id: null 2026-03-20 15:35:45.037432 | controller | qos_policy_id: null 2026-03-20 15:35:45.037534 | controller | security_group_ids: [] 2026-03-20 15:35:45.037628 | controller | status: DOWN 2026-03-20 15:35:45.037729 | controller | tags: [] 2026-03-20 15:35:45.037826 | controller | trunk_details: null 2026-03-20 15:35:45.037914 | controller | trusted: null 2026-03-20 15:35:45.038023 | controller | updated_at: '2026-03-20T15:35:44Z' 2026-03-20 15:35:45.404453 | controller | changed 2026-03-20 15:35:45.410262 | 2026-03-20 15:35:45.410319 | TASK [Set common facts for further usage] 2026-03-20 15:35:45.495303 | controller | ok 2026-03-20 15:35:45.503341 | 2026-03-20 15:35:45.503427 | TASK [Create the trunk port] 2026-03-20 15:35:46.044455 | controller | skipping: Conditional result was False 2026-03-20 15:35:46.057581 | 2026-03-20 15:35:46.057784 | TASK [Avoid NetworkManager to automatically picked up the new interface] 2026-03-20 15:35:46.114587 | controller | skipping: Conditional result was False 2026-03-20 15:35:46.127078 | 2026-03-20 15:35:46.127207 | TASK [Refresh NetworkManager] 2026-03-20 15:35:46.204550 | controller | skipping: Conditional result was False 2026-03-20 15:35:46.217073 | 2026-03-20 15:35:46.217219 | TASK [Attach the port to the instance] 2026-03-20 15:35:54.279061 | controller | changed 2026-03-20 15:35:54.290974 | 2026-03-20 15:35:54.291103 | TASK [Wait till the interface exists] 2026-03-20 15:35:54.595037 | controller -> 38.102.83.159 | changed 2026-03-20 15:35:54.603458 | 2026-03-20 15:35:54.603535 | TASK [Prepare interface data to be used configuring trunk and vlan ifaces] 2026-03-20 15:35:54.740370 | controller | ok 2026-03-20 15:35:54.753312 | 2026-03-20 15:35:54.753440 | LOOP [Create VLANs ports and attach them to the trunk] 2026-03-20 15:35:54.794263 | 2026-03-20 15:35:54.794528 | LOOP [Prepare VLANs interface data to be used configuring] 2026-03-20 15:35:54.833566 | 2026-03-20 15:35:54.833850 | LOOP [Create NetworkManager configuration file for the trunk port] 2026-03-20 15:35:55.662077 | controller -> 38.102.83.159 | changed: 2026-03-20 15:35:55.662280 | controller -> 38.102.83.159 | { 2026-03-20 15:35:55.662305 | controller -> 38.102.83.159 | "key": "default", 2026-03-20 15:35:55.662323 | controller -> 38.102.83.159 | "value": { 2026-03-20 15:35:55.662338 | controller -> 38.102.83.159 | "connection": "ci-private-network", 2026-03-20 15:35:55.662354 | controller -> 38.102.83.159 | "gw": "192.168.122.1", 2026-03-20 15:35:55.662369 | controller -> 38.102.83.159 | "iface": "eth1", 2026-03-20 15:35:55.662384 | controller -> 38.102.83.159 | "ip": "192.168.122.11/24", 2026-03-20 15:35:55.662398 | controller -> 38.102.83.159 | "mac": "fa:16:3e:0d:21:04", 2026-03-20 15:35:55.662413 | controller -> 38.102.83.159 | "mtu": "1500" 2026-03-20 15:35:55.662428 | controller -> 38.102.83.159 | } 2026-03-20 15:35:55.662442 | controller -> 38.102.83.159 | } 2026-03-20 15:35:55.662472 | 2026-03-20 15:35:55.678133 | 2026-03-20 15:35:55.678310 | TASK [Refresh NetworkManager] 2026-03-20 15:35:56.467630 | controller -> 38.102.83.159 | changed 2026-03-20 15:35:56.485430 | 2026-03-20 15:35:56.485563 | TASK [Debug fetch IP routes] 2026-03-20 15:35:56.779800 | controller -> 38.102.83.159 | changed 2026-03-20 15:35:56.793522 | 2026-03-20 15:35:56.793657 | TASK [Debug IP routes] 2026-03-20 15:35:56.840233 | controller -> 38.102.83.159 | ok 2026-03-20 15:35:56.853227 | 2026-03-20 15:35:56.853355 | TASK [Create instance ports and attach if necessary] 2026-03-20 15:35:56.882159 | controller | ok 2026-03-20 15:35:56.907587 | controller | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/trusted/project_0/review.rdoproject.org/config/playbooks/crc/bootstrap-ci-network-instance-create-attach-port.yml 2026-03-20 15:35:56.915471 | 2026-03-20 15:35:56.915548 | TASK [Print the host networking data] 2026-03-20 15:35:56.961408 | controller | ok 2026-03-20 15:35:56.972952 | 2026-03-20 15:35:56.973076 | TASK [Create the private network parent port] 2026-03-20 15:36:00.153660 | controller | admin_state_up: true 2026-03-20 15:36:00.153911 | controller | allowed_address_pairs: [] 2026-03-20 15:36:00.154244 | controller | binding_host_id: null 2026-03-20 15:36:00.154561 | controller | binding_profile: {} 2026-03-20 15:36:00.154821 | controller | binding_vif_details: {} 2026-03-20 15:36:00.155048 | controller | binding_vif_type: null 2026-03-20 15:36:00.155357 | controller | binding_vnic_type: normal 2026-03-20 15:36:00.155616 | controller | created_at: '2026-03-20T15:35:59Z' 2026-03-20 15:36:00.155869 | controller | data_plane_status: null 2026-03-20 15:36:00.156117 | controller | description: '' 2026-03-20 15:36:00.156344 | controller | device_id: '' 2026-03-20 15:36:00.156545 | controller | device_owner: '' 2026-03-20 15:36:00.156784 | controller | device_profile: null 2026-03-20 15:36:00.157468 | controller | dns_assignment: 2026-03-20 15:36:00.157674 | controller | - fqdn: host-192-168-122-10.openstacklocal. 2026-03-20 15:36:00.157681 | controller | hostname: host-192-168-122-10 2026-03-20 15:36:00.157700 | controller | ip_address: 192.168.122.10 2026-03-20 15:36:00.157709 | controller | dns_domain: '' 2026-03-20 15:36:00.157893 | controller | dns_name: '' 2026-03-20 15:36:00.158095 | controller | extra_dhcp_opts: [] 2026-03-20 15:36:00.158622 | controller | fixed_ips: 2026-03-20 15:36:00.158868 | controller | - ip_address: 192.168.122.10 2026-03-20 15:36:00.158877 | controller | subnet_id: fbbf2376-14dd-4853-abbb-f430e1538dd0 2026-03-20 15:36:00.158884 | controller | hardware_offload_type: null 2026-03-20 15:36:00.159059 | controller | hints: '' 2026-03-20 15:36:00.159296 | controller | id: 971ec772-62a8-4bf1-9450-4da0866ed20e 2026-03-20 15:36:00.159505 | controller | ip_allocation: immediate 2026-03-20 15:36:00.159747 | controller | mac_address: fa:16:3e:02:5c:b8 2026-03-20 15:36:00.159996 | controller | name: crc-a34aaf21-bc48-4d4d-b0c3-48ef72d66bb9 2026-03-20 15:36:00.160245 | controller | network_id: adbac40f-789f-4953-a393-3ccea9f4523a 2026-03-20 15:36:00.160457 | controller | numa_affinity_policy: null 2026-03-20 15:36:00.160768 | controller | port_security_enabled: false 2026-03-20 15:36:00.161019 | controller | project_id: 4b633c451ac74233be3721a3635275e5 2026-03-20 15:36:00.161234 | controller | propagate_uplink_status: null 2026-03-20 15:36:00.161439 | controller | resource_request: null 2026-03-20 15:36:00.161713 | controller | revision_number: 1 2026-03-20 15:36:00.161950 | controller | qos_network_policy_id: null 2026-03-20 15:36:00.162184 | controller | qos_policy_id: null 2026-03-20 15:36:00.162480 | controller | security_group_ids: [] 2026-03-20 15:36:00.162680 | controller | status: DOWN 2026-03-20 15:36:00.162899 | controller | tags: [] 2026-03-20 15:36:00.163145 | controller | trunk_details: null 2026-03-20 15:36:00.163345 | controller | trusted: null 2026-03-20 15:36:00.163581 | controller | updated_at: '2026-03-20T15:35:59Z' 2026-03-20 15:36:00.520843 | controller | changed 2026-03-20 15:36:00.529029 | 2026-03-20 15:36:00.529117 | TASK [Set common facts for further usage] 2026-03-20 15:36:00.618166 | controller | ok 2026-03-20 15:36:00.625878 | 2026-03-20 15:36:00.625958 | TASK [Create the trunk port] 2026-03-20 15:36:01.165762 | controller | skipping: Conditional result was False 2026-03-20 15:36:01.180375 | 2026-03-20 15:36:01.180519 | TASK [Avoid NetworkManager to automatically picked up the new interface] 2026-03-20 15:36:01.270862 | controller | skipping: Conditional result was False 2026-03-20 15:36:01.284385 | 2026-03-20 15:36:01.284520 | TASK [Refresh NetworkManager] 2026-03-20 15:36:01.363753 | controller | skipping: Conditional result was False 2026-03-20 15:36:01.372794 | 2026-03-20 15:36:01.372886 | TASK [Attach the port to the instance] 2026-03-20 15:36:09.440900 | controller | changed 2026-03-20 15:36:09.449189 | 2026-03-20 15:36:09.449286 | TASK [Wait till the interface exists] 2026-03-20 15:36:09.773506 | controller -> 38.102.83.181 | changed 2026-03-20 15:36:09.795928 | 2026-03-20 15:36:09.796153 | TASK [Prepare interface data to be used configuring trunk and vlan ifaces] 2026-03-20 15:36:09.916899 | controller | ok 2026-03-20 15:36:09.926300 | 2026-03-20 15:36:09.926451 | LOOP [Create VLANs ports and attach them to the trunk] 2026-03-20 15:36:09.974672 | controller | skipping: Conditional result was False 2026-03-20 15:36:12.989427 | controller | skipping: Conditional result was False 2026-03-20 15:36:15.998220 | controller | skipping: Conditional result was False 2026-03-20 15:36:16.020074 | 2026-03-20 15:36:16.020241 | LOOP [Prepare VLANs interface data to be used configuring] 2026-03-20 15:36:16.139458 | controller | ok: 2026-03-20 15:36:16.140023 | controller | { 2026-03-20 15:36:16.140108 | controller | "key": "internal-api", 2026-03-20 15:36:16.140161 | controller | "value": { 2026-03-20 15:36:16.140217 | controller | "ip": "172.17.0.5" 2026-03-20 15:36:16.140256 | controller | } 2026-03-20 15:36:16.140291 | controller | } 2026-03-20 15:36:16.140349 | 2026-03-20 15:36:16.176952 | controller | ok: 2026-03-20 15:36:16.177086 | controller | { 2026-03-20 15:36:16.177156 | controller | "key": "storage", 2026-03-20 15:36:16.177214 | controller | "value": { 2026-03-20 15:36:16.177266 | controller | "ip": "172.18.0.5" 2026-03-20 15:36:16.177314 | controller | } 2026-03-20 15:36:16.177361 | controller | } 2026-03-20 15:36:16.207979 | controller | ok: 2026-03-20 15:36:16.208277 | controller | { 2026-03-20 15:36:16.208349 | controller | "key": "tenant", 2026-03-20 15:36:16.208391 | controller | "value": { 2026-03-20 15:36:16.208427 | controller | "ip": "172.19.0.5" 2026-03-20 15:36:16.208462 | controller | } 2026-03-20 15:36:16.208495 | controller | } 2026-03-20 15:36:16.239485 | 2026-03-20 15:36:16.239655 | LOOP [Create NetworkManager configuration file for the trunk port] 2026-03-20 15:36:17.205607 | controller -> 38.102.83.181 | changed: 2026-03-20 15:36:17.205989 | controller -> 38.102.83.181 | { 2026-03-20 15:36:17.206046 | controller -> 38.102.83.181 | "key": "default", 2026-03-20 15:36:17.206086 | controller -> 38.102.83.181 | "value": { 2026-03-20 15:36:17.206122 | controller -> 38.102.83.181 | "connection": "ci-private-network", 2026-03-20 15:36:17.206157 | controller -> 38.102.83.181 | "gw": "192.168.122.1", 2026-03-20 15:36:17.206191 | controller -> 38.102.83.181 | "iface": "ens7", 2026-03-20 15:36:17.206225 | controller -> 38.102.83.181 | "ip": "192.168.122.10/24", 2026-03-20 15:36:17.206258 | controller -> 38.102.83.181 | "mac": "fa:16:3e:02:5c:b8", 2026-03-20 15:36:17.206291 | controller -> 38.102.83.181 | "mtu": "1500" 2026-03-20 15:36:17.206324 | controller -> 38.102.83.181 | } 2026-03-20 15:36:17.206356 | controller -> 38.102.83.181 | } 2026-03-20 15:36:18.173850 | controller -> 38.102.83.181 | changed: 2026-03-20 15:36:18.173979 | controller -> 38.102.83.181 | { 2026-03-20 15:36:18.174011 | controller -> 38.102.83.181 | "key": "internal-api", 2026-03-20 15:36:18.174035 | controller -> 38.102.83.181 | "value": { 2026-03-20 15:36:18.174059 | controller -> 38.102.83.181 | "connection": "ci-private-network-20", 2026-03-20 15:36:18.174082 | controller -> 38.102.83.181 | "iface": "ens7.20", 2026-03-20 15:36:18.174104 | controller -> 38.102.83.181 | "ip": "172.17.0.5/24", 2026-03-20 15:36:18.174125 | controller -> 38.102.83.181 | "mac": "52:54:00:dc:d1:84", 2026-03-20 15:36:18.174148 | controller -> 38.102.83.181 | "mtu": "1496", 2026-03-20 15:36:18.174169 | controller -> 38.102.83.181 | "parent_iface": "ens7", 2026-03-20 15:36:18.174190 | controller -> 38.102.83.181 | "vlan": 20 2026-03-20 15:36:18.174211 | controller -> 38.102.83.181 | } 2026-03-20 15:36:18.174233 | controller -> 38.102.83.181 | } 2026-03-20 15:36:19.113461 | controller -> 38.102.83.181 | changed: 2026-03-20 15:36:19.113642 | controller -> 38.102.83.181 | { 2026-03-20 15:36:19.113753 | controller -> 38.102.83.181 | "key": "storage", 2026-03-20 15:36:19.113822 | controller -> 38.102.83.181 | "value": { 2026-03-20 15:36:19.113879 | controller -> 38.102.83.181 | "connection": "ci-private-network-21", 2026-03-20 15:36:19.113918 | controller -> 38.102.83.181 | "iface": "ens7.21", 2026-03-20 15:36:19.113953 | controller -> 38.102.83.181 | "ip": "172.18.0.5/24", 2026-03-20 15:36:19.113986 | controller -> 38.102.83.181 | "mac": "52:54:00:b1:e3:64", 2026-03-20 15:36:19.114018 | controller -> 38.102.83.181 | "mtu": "1496", 2026-03-20 15:36:19.114049 | controller -> 38.102.83.181 | "parent_iface": "ens7", 2026-03-20 15:36:19.114080 | controller -> 38.102.83.181 | "vlan": 21 2026-03-20 15:36:19.114110 | controller -> 38.102.83.181 | } 2026-03-20 15:36:19.114141 | controller -> 38.102.83.181 | } 2026-03-20 15:36:20.003290 | controller -> 38.102.83.181 | changed: 2026-03-20 15:36:20.003652 | controller -> 38.102.83.181 | { 2026-03-20 15:36:20.003785 | controller -> 38.102.83.181 | "key": "tenant", 2026-03-20 15:36:20.003838 | controller -> 38.102.83.181 | "value": { 2026-03-20 15:36:20.003882 | controller -> 38.102.83.181 | "connection": "ci-private-network-22", 2026-03-20 15:36:20.003925 | controller -> 38.102.83.181 | "iface": "ens7.22", 2026-03-20 15:36:20.003969 | controller -> 38.102.83.181 | "ip": "172.19.0.5/24", 2026-03-20 15:36:20.004013 | controller -> 38.102.83.181 | "mac": "52:54:00:67:98:3d", 2026-03-20 15:36:20.004059 | controller -> 38.102.83.181 | "mtu": "1496", 2026-03-20 15:36:20.004106 | controller -> 38.102.83.181 | "parent_iface": "ens7", 2026-03-20 15:36:20.004154 | controller -> 38.102.83.181 | "vlan": 22 2026-03-20 15:36:20.004201 | controller -> 38.102.83.181 | } 2026-03-20 15:36:20.004244 | controller -> 38.102.83.181 | } 2026-03-20 15:36:20.030564 | 2026-03-20 15:36:20.030823 | TASK [Refresh NetworkManager] 2026-03-20 15:36:20.877395 | controller -> 38.102.83.181 | changed 2026-03-20 15:36:20.895305 | 2026-03-20 15:36:20.895439 | TASK [Debug fetch IP routes] 2026-03-20 15:36:21.250365 | controller -> 38.102.83.181 | changed 2026-03-20 15:36:21.263980 | 2026-03-20 15:36:21.264107 | TASK [Debug IP routes] 2026-03-20 15:36:21.321940 | controller -> 38.102.83.181 | ok 2026-03-20 15:36:21.337096 | 2026-03-20 15:36:21.337225 | TASK [Display some data about network ports] 2026-03-20 15:36:23.537866 | controller | +--------------------------------------+-------------------------------------------------+-------------------+-------------------------------------------------------------------------------+--------+ 2026-03-20 15:36:23.643046 | controller | | ID | Name | MAC Address | Fixed IP Addresses | Status | 2026-03-20 15:36:23.643104 | controller | +--------------------------------------+-------------------------------------------------+-------------------+-------------------------------------------------------------------------------+--------+ 2026-03-20 15:36:23.643113 | controller | | 30e4281f-5f88-42bc-a0da-d9e23bfff189 | controller-ca1c710b-d9a5-466b-a8d0-123e17aad387 | fa:16:3e:0d:21:04 | ip_address='192.168.122.11', subnet_id='fbbf2376-14dd-4853-abbb-f430e1538dd0' | ACTIVE | 2026-03-20 15:36:23.643121 | controller | | 7216053b-8e66-424e-8740-f1b7ba3569d8 | | fa:16:3e:16:de:40 | | DOWN | 2026-03-20 15:36:23.643128 | controller | | 971ec772-62a8-4bf1-9450-4da0866ed20e | crc-a34aaf21-bc48-4d4d-b0c3-48ef72d66bb9 | fa:16:3e:02:5c:b8 | ip_address='192.168.122.10', subnet_id='fbbf2376-14dd-4853-abbb-f430e1538dd0' | ACTIVE | 2026-03-20 15:36:23.643135 | controller | +--------------------------------------+-------------------------------------------------+-------------------+-------------------------------------------------------------------------------+--------+ 2026-03-20 15:36:23.892283 | controller | changed 2026-03-20 15:36:23.904264 | 2026-03-20 15:36:23.904392 | LOOP [Display server configuration] 2026-03-20 15:36:26.602755 | controller | changed 2026-03-20 15:36:26.602966 | controller | changed: All items complete 2026-03-20 15:36:26.603007 | 2026-03-20 15:36:31.956921 | controller | changed 2026-03-20 15:36:31.976412 | 2026-03-20 15:36:31.976558 | TASK [Fetch underneath provider DNSs] 2026-03-20 15:36:31.999598 | controller | ok 2026-03-20 15:36:32.008657 | controller | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/trusted/project_0/review.rdoproject.org/config/playbooks/crc/bootstrap-ci-network-fetch-provider-dns.yml 2026-03-20 15:36:32.022489 | 2026-03-20 15:36:32.022568 | TASK [Fetch the DNS list of the default interface] 2026-03-20 15:36:32.379508 | controller | ok 2026-03-20 15:36:32.384637 | 2026-03-20 15:36:32.384720 | TASK [Set provider DNS for further usage] 2026-03-20 15:36:32.456679 | controller | ok 2026-03-20 15:36:32.461780 | 2026-03-20 15:36:32.461838 | LOOP [Save networking data to file for further usage] 2026-03-20 15:36:33.409410 | controller -> 38.102.83.181 | changed: "crc" 2026-03-20 15:36:33.982449 | controller -> 38.102.83.159 | changed: "controller" 2026-03-20 15:36:34.005266 | 2026-03-20 15:36:34.005452 | TASK [Remove cloud_secrets file] 2026-03-20 15:36:34.279841 | controller | changed 2026-03-20 15:36:34.324201 | 2026-03-20 15:36:34.324343 | PLAY RECAP 2026-03-20 15:36:34.324407 | controller | ok: 48 changed: 22 unreachable: 0 failed: 0 skipped: 14 rescued: 0 ignored: 0 2026-03-20 15:36:34.324436 | 2026-03-20 15:36:34.445235 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/crc/bootstrap-ci-network.yml@master] 2026-03-20 15:36:34.449788 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/crc/prepare-crc.yaml@master] 2026-03-20 15:36:35.025527 | 2026-03-20 15:36:35.025714 | PLAY [Prefer ipv4 over ipv6] 2026-03-20 15:36:35.061048 | 2026-03-20 15:36:35.061144 | TASK [Configure /etc/gai.conf to prever ipv4 over ipv6] 2026-03-20 15:36:35.105942 | controller | skipping: Conditional result was False 2026-03-20 15:36:35.126752 | crc | skipping: Conditional result was False 2026-03-20 15:36:35.171999 | 2026-03-20 15:36:35.172073 | PLAY [Generic task for migration time to crc-cloud] 2026-03-20 15:36:35.191321 | 2026-03-20 15:36:35.191387 | TASK [Check if there is a image build date file] 2026-03-20 15:36:35.646012 | crc | ok 2026-03-20 15:36:35.662308 | 2026-03-20 15:36:35.662443 | TASK [Print image date] 2026-03-20 15:36:36.143671 | crc | Fri Mar 20 05:02:30 UTC 2026 | Image type: crc-cloud 2026-03-20 15:36:36.249483 | crc | ok 2026-03-20 15:36:36.264510 | 2026-03-20 15:36:36.264647 | TASK [Run crc-cloud preparation] 2026-03-20 15:36:36.307009 | crc | ok 2026-03-20 15:36:36.349689 | 2026-03-20 15:36:36.349805 | TASK [prepare-crc-cloud : Print build date] 2026-03-20 15:36:36.384171 | crc | ok 2026-03-20 15:36:36.394996 | crc | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/build_date.yaml 2026-03-20 15:36:36.408112 | 2026-03-20 15:36:36.408201 | TASK [prepare-crc-cloud : Check if there is a image build date file] 2026-03-20 15:36:36.690819 | crc | ok 2026-03-20 15:36:36.706886 | 2026-03-20 15:36:36.707043 | TASK [prepare-crc-cloud : Print image date if available] 2026-03-20 15:36:37.075262 | crc | Fri Mar 20 05:02:30 UTC 2026 | Image type: crc-cloud 2026-03-20 15:36:37.263307 | crc | ok 2026-03-20 15:36:37.279530 | 2026-03-20 15:36:37.279709 | TASK [prepare-crc-cloud : Tune Kubelet config file] 2026-03-20 15:36:37.307271 | crc | skipping: Conditional result was False 2026-03-20 15:36:37.324996 | 2026-03-20 15:36:37.325134 | TASK [prepare-crc-cloud : Reboot CRC host to get the network interfaces] 2026-03-20 15:36:37.366210 | crc | ok 2026-03-20 15:36:37.383525 | crc | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/reboot_crc.yaml 2026-03-20 15:36:37.398905 | 2026-03-20 15:36:37.398992 | TASK [prepare-crc-cloud : Reboot host after adding pull secret and changing kubelet node params] 2026-03-20 15:37:13.777149 | crc | changed 2026-03-20 15:37:13.791973 | 2026-03-20 15:37:13.792096 | TASK [Start Zuul console after reboot] 2026-03-20 15:37:13.836132 | crc | ok 2026-03-20 15:37:13.872263 | 2026-03-20 15:37:13.872487 | TASK [start-zuul-console : Start zuul_console daemon.] 2026-03-20 15:37:14.592611 | crc | ok 2026-03-20 15:37:14.620855 | 2026-03-20 15:37:14.621008 | TASK [prepare-crc-cloud : Prepare for executing crc-cloud] 2026-03-20 15:37:14.657533 | crc | ok 2026-03-20 15:37:14.668744 | crc | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/pre.yaml 2026-03-20 15:37:14.684185 | 2026-03-20 15:37:14.684268 | TASK [prepare-crc-cloud : Disable overwriting /etc/resolv.conf by the NetworkManager] 2026-03-20 15:37:15.726296 | crc | changed 2026-03-20 15:37:15.743252 | 2026-03-20 15:37:15.743551 | TASK [prepare-crc-cloud : Reload NetworkManager after creating custom rules] 2026-03-20 15:37:16.758803 | crc | changed 2026-03-20 15:37:16.779818 | 2026-03-20 15:37:16.779962 | TASK [prepare-crc-cloud : Inject pull secret into /var/lib/kubelet/config.json] 2026-03-20 15:37:17.805195 | crc | changed 2026-03-20 15:37:17.822323 | 2026-03-20 15:37:17.822484 | TASK [prepare-crc-cloud : Check if etcd slow profile set] 2026-03-20 15:37:17.852799 | crc | skipping: Conditional result was False 2026-03-20 15:37:17.873115 | 2026-03-20 15:37:17.873282 | TASK [prepare-crc-cloud : Use ramdisk for etcd] 2026-03-20 15:37:17.921298 | crc | ok 2026-03-20 15:37:17.940511 | crc | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/ramdisk.yaml 2026-03-20 15:37:18.010104 | 2026-03-20 15:37:18.010306 | TASK [prepare-crc-cloud : Move current etcd to other place] 2026-03-20 15:37:18.554542 | crc | ok 2026-03-20 15:37:18.575866 | 2026-03-20 15:37:18.576121 | TASK [prepare-crc-cloud : Mount ramdisk] 2026-03-20 15:37:18.999293 | crc | changed 2026-03-20 15:37:19.014745 | 2026-03-20 15:37:19.014884 | TASK [prepare-crc-cloud : Set proper permissions after mount] 2026-03-20 15:37:19.463029 | crc | changed 2026-03-20 15:37:19.480399 | 2026-03-20 15:37:19.480542 | TASK [prepare-crc-cloud : Copy content from old location to ramdisk] 2026-03-20 15:37:20.845897 | crc | changed 2026-03-20 15:37:20.854062 | 2026-03-20 15:37:20.854132 | TASK [prepare-crc-cloud : Set proper SELinux context] 2026-03-20 15:37:21.398067 | crc | ok 2026-03-20 15:37:21.415458 | 2026-03-20 15:37:21.415736 | TASK [prepare-crc-cloud : Block crc.testing domains] 2026-03-20 15:37:21.454598 | crc | ok 2026-03-20 15:37:21.466804 | crc | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/block_crc_domains.yaml 2026-03-20 15:37:21.481626 | 2026-03-20 15:37:21.481713 | LOOP [prepare-crc-cloud : Set crc.testing domains into /etc/hosts] 2026-03-20 15:37:22.021162 | crc | changed: line added 2026-03-20 15:37:22.275407 | crc | changed: line added 2026-03-20 15:37:22.514549 | crc | changed: line added 2026-03-20 15:37:22.798322 | crc | changed: line added 2026-03-20 15:37:23.082976 | crc | changed: line added 2026-03-20 15:37:23.383599 | crc | changed: line added 2026-03-20 15:37:23.689970 | crc | changed: line added 2026-03-20 15:37:23.986708 | crc | changed: line added 2026-03-20 15:37:24.015129 | 2026-03-20 15:37:24.015369 | TASK [prepare-crc-cloud : Get default interface name] 2026-03-20 15:37:24.413964 | crc | ens3 2026-03-20 15:37:24.638233 | crc | ok 2026-03-20 15:37:24.656903 | 2026-03-20 15:37:24.657116 | LOOP [prepare-crc-cloud : Block CRC domains also using iptables] 2026-03-20 15:37:25.169996 | crc | ok 2026-03-20 15:37:25.170833 | crc | changed: All items complete 2026-03-20 15:37:25.170957 | 2026-03-20 15:37:25.438540 | crc | ok 2026-03-20 15:37:25.727020 | crc | ok 2026-03-20 15:37:26.046779 | crc | ok 2026-03-20 15:37:26.337300 | crc | ok 2026-03-20 15:37:26.643788 | crc | ok 2026-03-20 15:37:26.948792 | crc | ok 2026-03-20 15:37:27.210050 | crc | ok 2026-03-20 15:37:27.234995 | 2026-03-20 15:37:27.235133 | TASK [prepare-crc-cloud : Set required facts] 2026-03-20 15:37:27.281431 | crc | ok 2026-03-20 15:37:27.299107 | crc | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/set_vars.yaml 2026-03-20 15:37:27.319525 | 2026-03-20 15:37:27.319610 | TASK [prepare-crc-cloud : Set required facts] 2026-03-20 15:37:27.400982 | crc | Output suppressed because no_log was given 2026-03-20 15:37:27.415327 | 2026-03-20 15:37:27.415457 | TASK [Start crc-cloud role when Zuul defined] 2026-03-20 15:37:27.483995 | crc | ok 2026-03-20 15:37:27.560780 | 2026-03-20 15:37:27.561012 | TASK [deploy-crc-cloud : Check if pull-secret is provided] 2026-03-20 15:37:27.588804 | crc | skipping: Conditional result was False 2026-03-20 15:37:27.609113 | 2026-03-20 15:37:27.609414 | TASK [deploy-crc-cloud : Create kubeconfig] 2026-03-20 15:37:27.661350 | crc | ok 2026-03-20 15:37:27.690002 | crc | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/kubeconfig.yaml 2026-03-20 15:37:27.711146 | 2026-03-20 15:37:27.711269 | TASK [deploy-crc-cloud : Create kube directory] 2026-03-20 15:37:28.032094 | crc | changed 2026-03-20 15:37:28.047361 | 2026-03-20 15:37:28.047491 | TASK [deploy-crc-cloud : Copy kubeconfig to user dir] 2026-03-20 15:37:28.498857 | crc | changed 2026-03-20 15:37:28.514098 | 2026-03-20 15:37:28.514229 | TASK [deploy-crc-cloud : Setup dnsmasq] 2026-03-20 15:37:28.555541 | crc | ok 2026-03-20 15:37:28.578478 | crc | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/dnsmasq.yaml 2026-03-20 15:37:28.596183 | 2026-03-20 15:37:28.596269 | TASK [deploy-crc-cloud : Disable overwriting /etc/resolv.conf by the NetworkManager] 2026-03-20 15:37:29.417172 | crc | ok 2026-03-20 15:37:29.434248 | 2026-03-20 15:37:29.434384 | TASK [deploy-crc-cloud : Restart NetworkManager when its needed] 2026-03-20 15:37:29.463466 | crc | skipping: Conditional result was False 2026-03-20 15:37:29.481268 | 2026-03-20 15:37:29.481409 | TASK [deploy-crc-cloud : Ensure /etc/resolv.conf is not a symlink and is writable] 2026-03-20 15:37:29.781486 | crc | ok 2026-03-20 15:37:29.797074 | 2026-03-20 15:37:29.797214 | TASK [deploy-crc-cloud : Create crc-dnsmasq.conf] 2026-03-20 15:37:30.583705 | crc | changed 2026-03-20 15:37:30.593363 | 2026-03-20 15:37:30.593445 | TASK [deploy-crc-cloud : Add domains into crc-dnsmasq.conf when alternative_domain] 2026-03-20 15:37:30.621165 | crc | skipping: Conditional result was False 2026-03-20 15:37:30.634448 | 2026-03-20 15:37:30.634566 | LOOP [deploy-crc-cloud : Set this host as first nameserver in /etc/resolv.conf] 2026-03-20 15:37:30.994557 | crc | changed: line replaced 2026-03-20 15:37:31.237792 | crc | ok 2026-03-20 15:37:31.498485 | crc | ok 2026-03-20 15:37:31.521487 | 2026-03-20 15:37:31.521620 | TASK [deploy-crc-cloud : Restart dnsmasq] 2026-03-20 15:37:32.435583 | crc | changed 2026-03-20 15:37:32.457615 | 2026-03-20 15:37:32.457790 | TASK [deploy-crc-cloud : Start kubelet] 2026-03-20 15:37:32.499740 | crc | ok 2026-03-20 15:37:32.519530 | crc | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/kubelet.yaml 2026-03-20 15:37:32.541203 | 2026-03-20 15:37:32.541284 | TASK [deploy-crc-cloud : Start and enable kubelet] 2026-03-20 15:37:51.262296 | crc | changed 2026-03-20 15:37:51.281590 | 2026-03-20 15:37:51.281747 | TASK [deploy-crc-cloud : Wait for port 6443 to be up] 2026-03-20 15:38:51.790297 | crc | ok 2026-03-20 15:38:51.806917 | 2026-03-20 15:38:51.807055 | TASK [deploy-crc-cloud : Wait for API to start before continue] 2026-03-20 15:38:53.270418 | crc | NAMESPACE NAME READY STATUS RESTARTS AGE 2026-03-20 15:38:53.270581 | crc | openshift-network-console networking-console-plugin-85b44fc459-gdk6g 1/1 Running 2 390d 2026-03-20 15:38:53.270602 | crc | openshift-network-diagnostics network-check-source-55646444c4-trplf 1/1 Running 2 390d 2026-03-20 15:38:53.270618 | crc | openshift-network-diagnostics network-check-target-xd92c 1/1 Running 2 390d 2026-03-20 15:38:53.270635 | crc | openshift-network-node-identity network-node-identity-vrzqb 2/2 Running 7 390d 2026-03-20 15:38:53.270658 | crc | openshift-network-operator iptables-alerter-4ln5h 1/1 Running 2 390d 2026-03-20 15:38:53.281255 | crc | openshift-network-operator network-operator-58b4c7f79c-55gtf 1/1 Running 4 390d 2026-03-20 15:38:53.416190 | crc | ok 2026-03-20 15:38:53.431721 | 2026-03-20 15:38:53.431862 | TASK [deploy-crc-cloud : Replace default pubkey] 2026-03-20 15:38:53.475859 | crc | ok 2026-03-20 15:38:53.497285 | crc | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/pubkey.yaml 2026-03-20 15:38:53.521929 | 2026-03-20 15:38:53.522039 | TASK [deploy-crc-cloud : Check if id_rsa.pub exists] 2026-03-20 15:38:53.829841 | crc | ok 2026-03-20 15:38:53.845271 | 2026-03-20 15:38:53.845409 | TASK [deploy-crc-cloud : Read pub key] 2026-03-20 15:38:54.404927 | crc | skipping: Conditional result was False 2026-03-20 15:38:54.421784 | 2026-03-20 15:38:54.421925 | TASK [deploy-crc-cloud : Wait for machineconfig] 2026-03-20 15:38:54.449717 | crc | skipping: Conditional result was False 2026-03-20 15:38:54.467719 | 2026-03-20 15:38:54.467870 | TASK [deploy-crc-cloud : Patch machineconfig 99-master-ssh] 2026-03-20 15:38:55.026968 | crc | skipping: Conditional result was False 2026-03-20 15:38:55.044995 | 2026-03-20 15:38:55.045139 | TASK [deploy-crc-cloud : Accept certificate] 2026-03-20 15:38:55.129750 | crc | ok 2026-03-20 15:38:55.149585 | crc | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/accept_cert.yaml 2026-03-20 15:38:55.182208 | 2026-03-20 15:38:55.182298 | LOOP [deploy-crc-cloud : Re-run accept certificates in Pending state few times] 2026-03-20 15:38:55.727393 | crc | ok 2026-03-20 15:38:55.727867 | crc | changed: All items complete 2026-03-20 15:38:55.727931 | 2026-03-20 15:39:06.355147 | crc | ok 2026-03-20 15:39:16.870037 | crc | ok 2026-03-20 15:39:16.941579 | 2026-03-20 15:39:16.941762 | TASK [deploy-crc-cloud : Create service account for - auto-csr-approver] 2026-03-20 15:39:17.816165 | crc | changed 2026-03-20 15:39:17.832618 | 2026-03-20 15:39:17.832778 | TASK [deploy-crc-cloud : Apply cluster cert approver] 2026-03-20 15:39:18.388430 | crc | serviceaccount/csr-approver-sa created 2026-03-20 15:39:18.395425 | crc | clusterrole.rbac.authorization.k8s.io/csr-approver created 2026-03-20 15:39:18.408140 | crc | clusterrolebinding.rbac.authorization.k8s.io/csr-approver-binding created 2026-03-20 15:39:18.420561 | crc | cronjob.batch/auto-csr-approver created 2026-03-20 15:39:18.889315 | crc | ok 2026-03-20 15:39:18.900340 | 2026-03-20 15:39:18.900431 | TASK [deploy-crc-cloud : Wait for cluster become healthy] 2026-03-20 15:39:18.935883 | crc | ok 2026-03-20 15:39:18.948585 | crc | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/wait_cluster_become_healthy.yaml 2026-03-20 15:39:18.979859 | 2026-03-20 15:39:18.979985 | TASK [deploy-crc-cloud : Check if specified OpenShift components are healthy etcd|openshift-apiserver] 2026-03-20 15:39:19.456888 | crc | True 2026-03-20 15:39:19.457696 | crc | False 2026-03-20 15:41:54.922857 | crc | changed 2026-03-20 15:41:54.939162 | 2026-03-20 15:41:54.939292 | TASK [deploy-crc-cloud : Output success message if components are healthy] 2026-03-20 15:41:54.994229 | crc | ok: OpenShift cluster components have become healthy in approximately 60 * 15 seconds. 2026-03-20 15:41:55.011694 | 2026-03-20 15:41:55.011859 | TASK [deploy-crc-cloud : Fail if any components are still unhealthy] 2026-03-20 15:41:55.037341 | crc | skipping: Conditional result was False 2026-03-20 15:41:55.091287 | 2026-03-20 15:41:55.091425 | TASK [deploy-crc-cloud : Set credentials] 2026-03-20 15:41:55.122926 | crc | ok 2026-03-20 15:41:55.134949 | crc | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/set_credentials.yaml 2026-03-20 15:41:55.158545 | 2026-03-20 15:41:55.158685 | TASK [deploy-crc-cloud : Ensure htpasswd.txt does not exists] 2026-03-20 15:41:55.448408 | crc | ok 2026-03-20 15:41:55.458627 | 2026-03-20 15:41:55.458739 | TASK [deploy-crc-cloud : Create temporary directory] 2026-03-20 15:41:55.857110 | crc | changed 2026-03-20 15:41:55.863946 | 2026-03-20 15:41:55.864021 | LOOP [deploy-crc-cloud : Get htpasswd] 2026-03-20 15:41:55.953357 | crc | Output suppressed because no_log was given 2026-03-20 15:41:55.972407 | crc | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/get_htpasswd.yaml 2026-03-20 15:41:55.974400 | crc | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/get_htpasswd.yaml 2026-03-20 15:41:55.976384 | crc | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/get_htpasswd.yaml 2026-03-20 15:41:55.995503 | 2026-03-20 15:41:55.995603 | TASK [deploy-crc-cloud : Get htpasswd for developer] 2026-03-20 15:42:14.022429 | crc | time="2026-03-20T15:42:14Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-03-20 15:42:14.028293 | crc | Trying to pull ghcr.io/crc-org/crc-cloud/httpd-tools:latest... 2026-03-20 15:42:14.390749 | crc | Getting image source signatures 2026-03-20 15:42:14.392038 | crc | Copying blob sha256:1730cf1a00b8c06596486f3456376e7660a42976bb9fcd6604ae22a70fca6b65 2026-03-20 15:42:14.392110 | crc | Copying blob sha256:f391d299bb65afd981883c104f434a04ee3ffb247e341247171779ef526ed65f 2026-03-20 15:42:17.233993 | crc | Copying config sha256:d81064f8940a06af9e1a39fc510c6bfc299bb166f99ee6a892f22ec84ee58041 2026-03-20 15:42:17.257045 | crc | Writing manifest to image destination 2026-03-20 15:42:18.076212 | crc | changed 2026-03-20 15:42:18.137475 | 2026-03-20 15:42:18.137649 | TASK [deploy-crc-cloud : Get htpasswd for kubeadmin] 2026-03-20 15:42:18.551861 | crc | time="2026-03-20T15:42:18Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-03-20 15:42:19.183317 | crc | changed 2026-03-20 15:42:19.201598 | 2026-03-20 15:42:19.201764 | TASK [deploy-crc-cloud : Get htpasswd for redhat] 2026-03-20 15:42:19.852111 | crc | time="2026-03-20T15:42:19Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly" 2026-03-20 15:42:20.770443 | crc | changed 2026-03-20 15:42:20.778351 | 2026-03-20 15:42:20.778410 | TASK [deploy-crc-cloud : Cleanup htpasswd.txt file] 2026-03-20 15:42:21.319506 | crc | changed 2026-03-20 15:42:21.335352 | 2026-03-20 15:42:21.335483 | TASK [deploy-crc-cloud : Create secret with generic htpass-secret] 2026-03-20 15:42:22.400446 | crc | ok 2026-03-20 15:42:22.416409 | 2026-03-20 15:42:22.416544 | TASK [deploy-crc-cloud : Replace htpass-secret] 2026-03-20 15:42:22.951892 | crc | secret/htpass-secret replaced 2026-03-20 15:42:23.019464 | crc | ok 2026-03-20 15:42:23.034768 | 2026-03-20 15:42:23.034899 | TASK [deploy-crc-cloud : Remove temporary directory] 2026-03-20 15:42:23.342352 | crc | changed 2026-03-20 15:42:23.357510 | 2026-03-20 15:42:23.357643 | TASK [deploy-crc-cloud : Replace default CA] 2026-03-20 15:42:23.399303 | crc | ok 2026-03-20 15:42:23.420622 | crc | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/replace_default_ca.yaml 2026-03-20 15:42:23.489863 | 2026-03-20 15:42:23.489997 | TASK [deploy-crc-cloud : Generate key] 2026-03-20 15:42:25.034440 | crc | changed 2026-03-20 15:42:25.051742 | 2026-03-20 15:42:25.051973 | TASK [deploy-crc-cloud : Generate cert] 2026-03-20 15:42:25.617248 | crc | changed 2026-03-20 15:42:25.632703 | 2026-03-20 15:42:25.632840 | TASK [deploy-crc-cloud : Generate csr] 2026-03-20 15:42:26.300922 | crc | .......+...+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*............+..+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.+.....+....+..............+.......+...+..+....+...+......+..+...+.........+.+.....+.........+..................+...+.+.....+...+...+....+.........+..+...+................+.........+.........+...+.....+...+....+...+..+............+.+...+..+...+............+.......+...+..................+.........+..+....+..............+............+.........+......+....+.........+.....+...+...+......+......+..........+.........+........+.+.....+.+......+.....+...+.+...+.........................................+..........+.........+.....+.+......+...+.....+......................+.....+................+.....+...+.........+......+....+..+.+.....+.......+..+...+.+...........+....+.....+.......+...+...........+...+...............+......+.+..................+...+.....+...+......+.+..+......................+......+...+......+...+...........+.+.....+...+...+...+......+...+.......+...+......+............+.........+..............+.+...+........+...+.+.....+.+..+.+.....+.+...........+...+.......+.....+.......+..+....+...........+.+.....+....+.........+.....+.........+................+.....+.+........+...+....+........+......+.+...........+.............+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2026-03-20 15:42:26.370391 | crc | ....+.......+.................+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+...+.....+.......+......+..+.+..+...+............+...+............+..........+...+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.................+.........+.+...+........+....+...............+......+...+..+...+......+...+......+....+..+.+.....+............+.......+...+......+..+....+........................+.....+......+...+...+...+......+...+......+....+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2026-03-20 15:42:26.374193 | crc | ----- 2026-03-20 15:42:26.698276 | crc | changed 2026-03-20 15:42:26.715198 | 2026-03-20 15:42:26.715378 | TASK [deploy-crc-cloud : Generate user cert] 2026-03-20 15:42:27.073657 | crc | Certificate request self-signature ok 2026-03-20 15:42:27.084500 | crc | subject=O = ${GROUP}, CN = core 2026-03-20 15:42:27.283171 | crc | changed 2026-03-20 15:42:27.298842 | 2026-03-20 15:42:27.298976 | TASK [deploy-crc-cloud : Create configmap] 2026-03-20 15:42:27.877422 | crc | configmap/client-ca-custom created 2026-03-20 15:42:28.373052 | crc | ok 2026-03-20 15:42:28.390442 | 2026-03-20 15:42:28.390579 | TASK [deploy-crc-cloud : Patch apiserver] 2026-03-20 15:42:29.004748 | crc | apiserver.config.openshift.io/cluster patched 2026-03-20 15:42:29.474067 | crc | ok 2026-03-20 15:42:29.489062 | 2026-03-20 15:42:29.489188 | TASK [deploy-crc-cloud : Create configmap] 2026-03-20 15:42:29.959742 | crc | W0320 15:42:29.959180 13814 helpers.go:703] --dry-run is deprecated and can be replaced with --dry-run=client. 2026-03-20 15:42:29.993256 | crc | configmap/admin-kubeconfig-client-ca replaced 2026-03-20 15:42:30.059359 | crc | ok 2026-03-20 15:42:30.075401 | 2026-03-20 15:42:30.075535 | TASK [deploy-crc-cloud : Login to the OpenShift cluster] 2026-03-20 15:42:30.118818 | crc | ok 2026-03-20 15:42:30.135677 | crc | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/login.yaml 2026-03-20 15:42:30.173273 | 2026-03-20 15:42:30.173335 | TASK [deploy-crc-cloud : Try to login after all changes] 2026-03-20 15:42:30.613682 | crc | WARNING: Using insecure TLS client config. Setting this option is not supported! 2026-03-20 15:42:30.748534 | crc | 2026-03-20 15:42:30.748586 | crc | Login failed (401 Unauthorized) 2026-03-20 15:42:30.753262 | crc | Verify you have provided the correct credentials. 2026-03-20 15:44:03.668524 | crc | ok 2026-03-20 15:44:03.683855 | 2026-03-20 15:44:03.683988 | TASK [deploy-crc-cloud : Patch pull secret] 2026-03-20 15:44:03.725452 | crc | ok 2026-03-20 15:44:03.742476 | crc | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/patch_pull_secret.yaml 2026-03-20 15:44:03.783505 | 2026-03-20 15:44:03.783582 | TASK [deploy-crc-cloud : Patch pull secret] 2026-03-20 15:44:04.859324 | crc | Output suppressed because no_log was given 2026-03-20 15:44:04.876008 | 2026-03-20 15:44:04.876162 | TASK [deploy-crc-cloud : Wait for cluster become healthy after patching CA and pull secret] 2026-03-20 15:44:04.918265 | crc | ok 2026-03-20 15:44:04.936197 | crc | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/wait_cluster_become_healthy.yaml 2026-03-20 15:44:04.982634 | 2026-03-20 15:44:04.982791 | TASK [deploy-crc-cloud : Check if specified OpenShift components are healthy etcd|openshift-apiserver] 2026-03-20 15:44:05.493338 | crc | True 2026-03-20 15:44:05.495231 | crc | True 2026-03-20 15:44:05.541083 | crc | changed 2026-03-20 15:44:05.556506 | 2026-03-20 15:44:05.556866 | TASK [deploy-crc-cloud : Output success message if components are healthy] 2026-03-20 15:44:05.610074 | crc | ok: OpenShift cluster components have become healthy in approximately 60 * 15 seconds. 2026-03-20 15:44:05.625868 | 2026-03-20 15:44:05.626004 | TASK [deploy-crc-cloud : Fail if any components are still unhealthy] 2026-03-20 15:44:05.652840 | crc | skipping: Conditional result was False 2026-03-20 15:44:05.670086 | 2026-03-20 15:44:05.670231 | TASK [deploy-crc-cloud : Create certificate and patch secret] 2026-03-20 15:44:05.697522 | crc | skipping: Conditional result was False 2026-03-20 15:44:05.714972 | 2026-03-20 15:44:05.715110 | TASK [deploy-crc-cloud : Wait for cluster become healthy after adding domain] 2026-03-20 15:44:05.758192 | crc | ok 2026-03-20 15:44:05.774197 | crc | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/wait_cluster_become_healthy.yaml 2026-03-20 15:44:05.813908 | 2026-03-20 15:44:05.813971 | TASK [deploy-crc-cloud : Check if specified OpenShift components are healthy etcd|openshift-apiserver] 2026-03-20 15:44:06.303196 | crc | True 2026-03-20 15:44:06.303848 | crc | True 2026-03-20 15:44:06.361903 | crc | changed 2026-03-20 15:44:06.376909 | 2026-03-20 15:44:06.377034 | TASK [deploy-crc-cloud : Output success message if components are healthy] 2026-03-20 15:44:06.428818 | crc | ok: OpenShift cluster components have become healthy in approximately 60 * 15 seconds. 2026-03-20 15:44:06.443486 | 2026-03-20 15:44:06.443617 | TASK [deploy-crc-cloud : Fail if any components are still unhealthy] 2026-03-20 15:44:06.471244 | crc | skipping: Conditional result was False 2026-03-20 15:44:06.488414 | 2026-03-20 15:44:06.488574 | TASK [deploy-crc-cloud : Patch ingress config] 2026-03-20 15:44:06.515558 | crc | skipping: Conditional result was False 2026-03-20 15:44:06.532407 | 2026-03-20 15:44:06.532538 | TASK [deploy-crc-cloud : Patch api server] 2026-03-20 15:44:06.559817 | crc | skipping: Conditional result was False 2026-03-20 15:44:06.577947 | 2026-03-20 15:44:06.578089 | TASK [deploy-crc-cloud : Patch default route] 2026-03-20 15:44:06.606825 | crc | skipping: Conditional result was False 2026-03-20 15:44:06.624616 | 2026-03-20 15:44:06.624786 | TASK [deploy-crc-cloud : Wait for cluster become healthy after changing ingress api server and default route] 2026-03-20 15:44:06.668881 | crc | ok 2026-03-20 15:44:06.687715 | crc | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/work/src/github.com/crc-org/crc-cloud/ansible/roles/deploy-crc-cloud/tasks/wait_cluster_become_healthy.yaml 2026-03-20 15:44:06.732203 | 2026-03-20 15:44:06.732270 | TASK [deploy-crc-cloud : Check if specified OpenShift components are healthy authentication|console|etcd|ingress|openshift-apiserver] 2026-03-20 15:44:07.249402 | crc | False 2026-03-20 15:44:07.249971 | crc | True 2026-03-20 15:44:07.250005 | crc | True 2026-03-20 15:44:07.250021 | crc | True 2026-03-20 15:44:07.250035 | crc | True 2026-03-20 15:45:24.430821 | crc | changed 2026-03-20 15:45:24.448514 | 2026-03-20 15:45:24.448753 | TASK [deploy-crc-cloud : Output success message if components are healthy] 2026-03-20 15:45:24.501122 | crc | ok: OpenShift cluster components have become healthy in approximately 60 * 15 seconds. 2026-03-20 15:45:24.516389 | 2026-03-20 15:45:24.516519 | TASK [deploy-crc-cloud : Fail if any components are still unhealthy] 2026-03-20 15:45:24.543436 | crc | skipping: Conditional result was False 2026-03-20 15:45:24.562910 | 2026-03-20 15:45:24.563053 | TASK [deploy-crc-cloud : Get console route] 2026-03-20 15:45:24.591443 | crc | skipping: Conditional result was False 2026-03-20 15:45:24.621957 | 2026-03-20 15:45:24.622045 | TASK [Start crc-cloud role when Zuul is not defined] 2026-03-20 15:45:24.667874 | crc | skipping: Conditional result was False 2026-03-20 15:45:24.684911 | 2026-03-20 15:45:24.685052 | TASK [prepare-crc-cloud : Recreate kubelet config.json] 2026-03-20 15:45:24.732653 | crc | ok 2026-03-20 15:45:24.753847 | crc | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/inject-pull-secret.yaml 2026-03-20 15:45:24.784912 | 2026-03-20 15:45:24.784998 | TASK [prepare-crc-cloud : Fail when openshift_pull_sec not provided] 2026-03-20 15:45:24.850648 | crc | skipping: Conditional result was False 2026-03-20 15:45:24.869601 | 2026-03-20 15:45:24.869768 | TASK [prepare-crc-cloud : Cleanup file] 2026-03-20 15:45:25.432752 | crc | Output suppressed because no_log was given 2026-03-20 15:45:25.447948 | 2026-03-20 15:45:25.448084 | TASK [prepare-crc-cloud : Copy pull-secret to kubelet config.json] 2026-03-20 15:45:25.855095 | crc | Output suppressed because no_log was given 2026-03-20 15:45:25.870213 | 2026-03-20 15:45:25.870355 | TASK [prepare-crc-cloud : Replace singleqoute with doublequote] 2026-03-20 15:45:26.373400 | crc | changed: 42 replacements made 2026-03-20 15:45:26.432801 | 2026-03-20 15:45:26.432953 | TASK [prepare-crc-cloud : Remove spaces] 2026-03-20 15:45:26.808290 | crc | changed: 20 replacements made 2026-03-20 15:45:26.823276 | 2026-03-20 15:45:26.823439 | TASK [prepare-crc-cloud : Ensure you are logged in to the OpenShift] 2026-03-20 15:45:26.866630 | crc | ok 2026-03-20 15:45:26.884781 | crc | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/openshift_login.yaml 2026-03-20 15:45:26.917633 | 2026-03-20 15:45:26.917779 | TASK [prepare-crc-cloud : Login to the OpenShift] 2026-03-20 15:45:27.391873 | crc | WARNING: Using insecure TLS client config. Setting this option is not supported! 2026-03-20 15:45:27.502111 | crc | 2026-03-20 15:45:27.502177 | crc | Login successful. 2026-03-20 15:45:27.520233 | crc | 2026-03-20 15:45:27.520286 | crc | You have access to 65 projects, the list has been suppressed. You can list all projects with 'oc projects' 2026-03-20 15:45:27.558756 | crc | 2026-03-20 15:45:27.558798 | crc | Using project "default". 2026-03-20 15:45:27.959565 | crc | ok 2026-03-20 15:45:27.976706 | 2026-03-20 15:45:27.976880 | TASK [prepare-crc-cloud : Post tasks after crc-cloud script execution] 2026-03-20 15:45:28.020351 | crc | ok 2026-03-20 15:45:28.043843 | crc | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/post.yaml 2026-03-20 15:45:28.078389 | 2026-03-20 15:45:28.078477 | LOOP [prepare-crc-cloud : Set DNS nameservers provided by cloud provider] 2026-03-20 15:45:28.401236 | crc | changed: line added 2026-03-20 15:45:28.746204 | crc | changed: line added 2026-03-20 15:45:28.766145 | 2026-03-20 15:45:28.766352 | TASK [prepare-crc-cloud : Increase dnsmasq cache size] 2026-03-20 15:45:29.126305 | crc | changed: line added 2026-03-20 15:45:29.141610 | 2026-03-20 15:45:29.141780 | TASK [prepare-crc-cloud : Reload NetworkManager after adding cache and DNS server] 2026-03-20 15:45:30.027581 | crc | changed 2026-03-20 15:45:30.048491 | 2026-03-20 15:45:30.048622 | TASK [prepare-crc-cloud : Ensure kube config does not exists] 2026-03-20 15:45:30.408468 | crc | ok 2026-03-20 15:45:30.424252 | 2026-03-20 15:45:30.424388 | TASK [prepare-crc-cloud : Copy new generated KUBECONFIG] 2026-03-20 15:45:30.452596 | crc | skipping: Conditional result was False 2026-03-20 15:45:30.469626 | 2026-03-20 15:45:30.469821 | TASK [prepare-crc-cloud : Delete all openshift-marketplace pods] 2026-03-20 15:45:30.953075 | crc | pod "certified-operators-b777z" deleted 2026-03-20 15:45:30.966496 | crc | pod "community-operators-wsh8b" deleted 2026-03-20 15:45:30.983265 | crc | pod "marketplace-operator-79b997595-crtz5" deleted 2026-03-20 15:45:30.999873 | crc | pod "redhat-marketplace-grn6l" deleted 2026-03-20 15:45:31.012940 | crc | pod "redhat-operators-wv9qh" deleted 2026-03-20 15:45:32.047023 | crc | ok 2026-03-20 15:45:32.063050 | 2026-03-20 15:45:32.063244 | TASK [prepare-crc-cloud : Create workaround for expired cert] 2026-03-20 15:45:32.143991 | crc | ok 2026-03-20 15:45:32.155782 | crc | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/cluster_cert.yaml 2026-03-20 15:45:32.184476 | 2026-03-20 15:45:32.184553 | LOOP [prepare-crc-cloud : Re-run accept certificates in Pending state few times] 2026-03-20 15:45:32.678162 | crc | ok 2026-03-20 15:45:32.678621 | crc | changed: All items complete 2026-03-20 15:45:32.678753 | 2026-03-20 15:45:43.120802 | crc | ok 2026-03-20 15:45:53.818721 | crc | ok 2026-03-20 15:45:53.841451 | 2026-03-20 15:45:53.841655 | TASK [prepare-crc-cloud : Create service account for - auto-csr-approver] 2026-03-20 15:45:55.856869 | crc | ok 2026-03-20 15:45:55.873161 | 2026-03-20 15:45:55.873376 | TASK [prepare-crc-cloud : Apply cluster cert approver] 2026-03-20 15:45:56.532282 | crc | serviceaccount/csr-approver-sa unchanged 2026-03-20 15:45:56.558029 | crc | clusterrole.rbac.authorization.k8s.io/csr-approver unchanged 2026-03-20 15:45:56.584519 | crc | clusterrolebinding.rbac.authorization.k8s.io/csr-approver-binding unchanged 2026-03-20 15:45:56.611785 | crc | cronjob.batch/auto-csr-approver unchanged 2026-03-20 15:45:56.936473 | crc | ok 2026-03-20 15:45:56.958984 | 2026-03-20 15:45:56.959250 | TASK [prepare-crc-cloud : Pause machineconfigpool] 2026-03-20 15:45:57.010744 | crc | ok 2026-03-20 15:45:57.031293 | crc | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/trusted/project_0/review.rdoproject.org/config/roles/prepare-crc-cloud/tasks/pause_mcp.yaml 2026-03-20 15:45:57.069952 | 2026-03-20 15:45:57.070022 | LOOP [prepare-crc-cloud : Pause the machineconfig configuration if mcp is not needed] 2026-03-20 15:45:57.672055 | crc | ok 2026-03-20 15:45:58.098305 | crc | ok 2026-03-20 15:45:58.121442 | 2026-03-20 15:45:58.121587 | TASK [prepare-crc-cloud : Print current machineconfigpool] 2026-03-20 15:45:58.605163 | crc | NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE 2026-03-20 15:45:58.605266 | crc | master rendered-master-f83a149c7490c3b0bf850df839837383 False True True 1 0 0 1 390d 2026-03-20 15:45:58.605293 | crc | worker rendered-worker-594cc07c9feefb5e37f2530ac457d36c True False False 0 0 0 0 390d 2026-03-20 15:45:58.685951 | crc | ok 2026-03-20 15:45:58.701488 | 2026-03-20 15:45:58.701637 | TASK [prepare-crc-cloud : Set low profile to etcd to avoid flakiness] 2026-03-20 15:45:58.742308 | crc | skipping: Conditional result was False 2026-03-20 15:45:58.761125 | 2026-03-20 15:45:58.761288 | TASK [prepare-crc-cloud : Fix machineconfigpool degradation] 2026-03-20 15:45:58.792914 | crc | skipping: Conditional result was False 2026-03-20 15:45:58.812364 | 2026-03-20 15:45:58.812506 | TASK [prepare-crc-cloud : Remove drain machineconfig for the node] 2026-03-20 15:45:58.843792 | crc | skipping: Conditional result was False 2026-03-20 15:45:58.862916 | 2026-03-20 15:45:58.863061 | TASK [prepare-crc-cloud : Post machineconfigpool tasks] 2026-03-20 15:45:58.902703 | crc | skipping: Conditional result was False 2026-03-20 15:45:58.922802 | 2026-03-20 15:45:58.922943 | TASK [prepare-crc-cloud : Enable monitoring and cluster-monitoring-operator] 2026-03-20 15:45:58.952968 | crc | skipping: Conditional result was False 2026-03-20 15:45:59.064387 | 2026-03-20 15:45:59.064570 | TASK [Run crc-extracted preparation] 2026-03-20 15:45:59.101139 | crc | skipping: Conditional result was False 2026-03-20 15:45:59.112427 | 2026-03-20 15:45:59.112514 | TASK [Fail when image build date does not exists] 2026-03-20 15:45:59.140385 | crc | skipping: Conditional result was False 2026-03-20 15:45:59.188781 | 2026-03-20 15:45:59.188974 | PLAY [Set IOPS and Read/Write disk limitation] 2026-03-20 15:45:59.226276 | 2026-03-20 15:45:59.226345 | TASK [Set limit for disk usage - IOPS and Read/Write] 2026-03-20 15:45:59.309248 | controller | ok 2026-03-20 15:45:59.328222 | crc | ok 2026-03-20 15:45:59.367798 | 2026-03-20 15:45:59.367908 | TASK [disk-usage-limitation : Get block device major:minor] 2026-03-20 15:45:59.872497 | controller | 252:0 2026-03-20 15:45:59.805049 | crc | 252:0 2026-03-20 15:45:59.937511 | crc | ok 2026-03-20 15:45:59.950368 | controller | ok 2026-03-20 15:45:59.964759 | 2026-03-20 15:45:59.964902 | LOOP [disk-usage-limitation : Ensure that some cgroup dirs exists] 2026-03-20 15:46:00.217142 | controller | ok: "/sys/fs/cgroup/init.scope" 2026-03-20 15:46:00.217653 | controller | ok: All items complete 2026-03-20 15:46:00.217838 | 2026-03-20 15:46:00.361870 | crc | ok: "/sys/fs/cgroup/init.scope" 2026-03-20 15:46:00.412837 | controller | changed: "/sys/fs/cgroup/machine.slice" 2026-03-20 15:46:00.627757 | crc | ok: "/sys/fs/cgroup/machine.slice" 2026-03-20 15:46:00.640150 | controller | ok: "/sys/fs/cgroup/system.slice" 2026-03-20 15:46:00.839537 | controller | ok: "/sys/fs/cgroup/user.slice" 2026-03-20 15:46:00.928521 | crc | ok: "/sys/fs/cgroup/system.slice" 2026-03-20 15:46:01.232164 | crc | ok: "/sys/fs/cgroup/user.slice" 2026-03-20 15:46:01.251434 | 2026-03-20 15:46:01.251746 | TASK [disk-usage-limitation : Ensure systemd configuration directory] 2026-03-20 15:46:01.516107 | controller | changed 2026-03-20 15:46:01.623416 | crc | ok 2026-03-20 15:46:01.636216 | 2026-03-20 15:46:01.636355 | TASK [disk-usage-limitation : Enable IO limit via override configuration] 2026-03-20 15:46:02.263921 | controller | changed 2026-03-20 15:46:02.572753 | crc | changed 2026-03-20 15:46:02.585007 | 2026-03-20 15:46:02.585135 | TASK [disk-usage-limitation : Flush handlers that will call systemd daemon-reload] 2026-03-20 15:46:02.607942 | 2026-03-20 15:46:02.608040 | TASK [disk-usage-limitation : Flush handlers that will call systemd daemon-reload] 2026-03-20 15:46:02.624000 | 2026-03-20 15:46:02.624092 | TASK [disk-usage-limitation : Reload systemd daemon after enabling IOAccounting] 2026-03-20 15:46:03.590508 | controller | ok 2026-03-20 15:46:04.736467 | crc | ok 2026-03-20 15:46:04.748695 | 2026-03-20 15:46:04.748861 | TASK [disk-usage-limitation : Wait for system.slice cgroup to have io.max file] 2026-03-20 15:46:05.044180 | controller | ok 2026-03-20 15:46:05.115589 | crc | ok 2026-03-20 15:46:05.123978 | 2026-03-20 15:46:05.124101 | LOOP [disk-usage-limitation : Set IOPS limit in the root cgroup] 2026-03-20 15:46:05.498842 | crc | ok 2026-03-20 15:46:05.501572 | crc | ok: All items complete 2026-03-20 15:46:05.501705 | 2026-03-20 15:46:05.514576 | controller | ok 2026-03-20 15:46:05.740437 | controller | ok 2026-03-20 15:46:05.784455 | crc | ok 2026-03-20 15:46:06.034961 | controller | ok 2026-03-20 15:46:06.065392 | crc | ok 2026-03-20 15:46:06.237852 | controller | ok 2026-03-20 15:46:06.392074 | crc | ok 2026-03-20 15:46:06.419456 | 2026-03-20 15:46:06.419741 | TASK [disk-usage-limitation : Verify the IOPS limit] 2026-03-20 15:46:06.766054 | controller | init 2026-03-20 15:46:06.766099 | controller | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000 2026-03-20 15:46:06.766106 | controller | machine 2026-03-20 15:46:06.766111 | controller | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000 2026-03-20 15:46:06.766116 | controller | system 2026-03-20 15:46:06.766120 | controller | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000 2026-03-20 15:46:06.766125 | controller | user 2026-03-20 15:46:06.766129 | controller | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000 2026-03-20 15:46:06.839743 | crc | init 2026-03-20 15:46:06.839828 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000 2026-03-20 15:46:06.839841 | crc | machine 2026-03-20 15:46:06.839850 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000 2026-03-20 15:46:06.839861 | crc | system 2026-03-20 15:46:06.842409 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000 2026-03-20 15:46:06.842815 | crc | user 2026-03-20 15:46:06.844149 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000 2026-03-20 15:46:07.023845 | controller | ok 2026-03-20 15:46:07.041570 | crc | ok 2026-03-20 15:46:07.055328 | 2026-03-20 15:46:07.055469 | TASK [disk-usage-limitation : Check if kubepods.slice exists] 2026-03-20 15:46:07.390048 | crc | ok 2026-03-20 15:46:07.406130 | controller | ok 2026-03-20 15:46:07.418652 | 2026-03-20 15:46:07.418848 | TASK [disk-usage-limitation : Set the limit] 2026-03-20 15:46:08.034001 | controller | skipping: Conditional result was False 2026-03-20 15:46:08.049961 | crc | ok 2026-03-20 15:46:08.063223 | 2026-03-20 15:46:08.063351 | TASK [disk-usage-limitation : Verify the IOPS limit - kubepods slice] 2026-03-20 15:46:08.450497 | crc | kubepods 2026-03-20 15:46:08.451681 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000 2026-03-20 15:46:08.658235 | controller | skipping: Conditional result was False 2026-03-20 15:46:08.675052 | crc | ok 2026-03-20 15:46:08.688323 | 2026-03-20 15:46:08.688469 | TASK [disk-usage-limitation : Print the current io.max value] 2026-03-20 15:46:08.768959 | controller | ok: 2026-03-20 15:46:08.769276 | controller | init 2026-03-20 15:46:08.769313 | controller | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000 2026-03-20 15:46:08.769340 | controller | machine 2026-03-20 15:46:08.769363 | controller | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000 2026-03-20 15:46:08.769386 | controller | system 2026-03-20 15:46:08.769410 | controller | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000 2026-03-20 15:46:08.769433 | controller | user 2026-03-20 15:46:08.769456 | controller | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000 2026-03-20 15:46:08.798912 | crc | ok: 2026-03-20 15:46:08.798980 | crc | init 2026-03-20 15:46:08.799014 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000 2026-03-20 15:46:08.799041 | crc | machine 2026-03-20 15:46:08.799064 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000 2026-03-20 15:46:08.799088 | crc | system 2026-03-20 15:46:08.799111 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000 2026-03-20 15:46:08.799134 | crc | user 2026-03-20 15:46:08.799157 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000 kubepods 2026-03-20 15:46:08.799179 | crc | 252:0 rbps=262144000 wbps=262144000 riops=18000 wiops=18000 2026-03-20 15:46:08.870648 | 2026-03-20 15:46:08.870778 | PLAY RECAP 2026-03-20 15:46:08.870830 | controller | ok: 10 changed: 3 unreachable: 0 failed: 0 skipped: 3 rescued: 0 ignored: 0 2026-03-20 15:46:08.870855 | crc | ok: 112 changed: 37 unreachable: 0 failed: 0 skipped: 28 rescued: 0 ignored: 0 2026-03-20 15:46:08.870873 | 2026-03-20 15:46:09.044191 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/crc/prepare-crc.yaml@master] 2026-03-20 15:46:09.049007 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/crc/podman.yaml@master] 2026-03-20 15:46:09.688925 | 2026-03-20 15:46:09.689053 | PLAY [all,!compute] 2026-03-20 15:46:09.723656 | 2026-03-20 15:46:09.723769 | TASK [Enable container-tools rhel-modules for c8/crc node] 2026-03-20 15:46:10.288312 | controller | skipping: Conditional result was False 2026-03-20 15:46:10.289451 | crc | skipping: Conditional result was False 2026-03-20 15:46:10.303334 | 2026-03-20 15:46:10.303529 | TASK [Install podman] 2026-03-20 15:46:10.407541 | crc | skipping: Conditional result was False 2026-03-20 15:47:05.454500 | controller | changed 2026-03-20 15:47:05.460718 | 2026-03-20 15:47:05.460780 | TASK [Set var name for quay login zuul secret] 2026-03-20 15:47:05.513153 | controller | skipping: Conditional result was False 2026-03-20 15:47:05.554397 | crc | skipping: Conditional result was False 2026-03-20 15:47:05.564418 | 2026-03-20 15:47:05.564529 | TASK [Print the username] 2026-03-20 15:47:06.056596 | controller | openstack-k8s-operators+cirobot 2026-03-20 15:47:06.158012 | crc | skipping: Conditional result was False 2026-03-20 15:47:06.167410 | controller | ok: Runtime: 0:00:00.003993 2026-03-20 15:47:06.175650 | 2026-03-20 15:47:06.175775 | TASK [Fail when user or password is not set] 2026-03-20 15:47:06.257426 | controller | skipping: Conditional result was False 2026-03-20 15:47:06.268793 | crc | skipping: Conditional result was False 2026-03-20 15:47:06.281765 | 2026-03-20 15:47:06.281898 | TASK [Set vars for quay login] 2026-03-20 15:47:06.363242 | controller | Output suppressed because no_log was given 2026-03-20 15:47:06.381048 | 2026-03-20 15:47:06.381137 | LOOP [Perform container registry login(s) with podman] 2026-03-20 15:47:07.172661 | controller | Output suppressed because no_log was given 2026-03-20 15:47:07.187394 | 2026-03-20 15:47:07.187535 | LOOP [Perform container registry login(s) with buildah] 2026-03-20 15:47:07.325901 | controller | Output suppressed because no_log was given 2026-03-20 15:47:07.326286 | 2026-03-20 15:47:07.350611 | 2026-03-20 15:47:07.350796 | TASK [Set Insecure registry for content provider] 2026-03-20 15:47:07.429949 | crc | skipping: Conditional result was False 2026-03-20 15:47:07.789656 | controller | changed: Block inserted 2026-03-20 15:47:07.865464 | 2026-03-20 15:47:07.865632 | PLAY RECAP 2026-03-20 15:47:07.865709 | controller | ok: 5 changed: 4 unreachable: 0 failed: 0 skipped: 4 rescued: 0 ignored: 0 2026-03-20 15:47:07.865739 | crc | ok: 0 changed: 0 unreachable: 0 failed: 0 skipped: 9 rescued: 0 ignored: 0 2026-03-20 15:47:07.865758 | 2026-03-20 15:47:08.000740 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/crc/podman.yaml@master] 2026-03-20 15:47:08.006190 | PRE-RUN START: [trusted : review.rdoproject.org/config/playbooks/data_plane_adoption/dpa_RH_subscription_registry_login.yaml@master] 2026-03-20 15:47:08.627559 | 2026-03-20 15:47:08.627723 | PLAY [rh-subscription] 2026-03-20 15:47:08.627829 | 2026-03-20 15:47:08.627862 | PLAY RECAP 2026-03-20 15:47:08.627890 | 2026-03-20 15:47:08.724819 | PRE-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/data_plane_adoption/dpa_RH_subscription_registry_login.yaml@master] 2026-03-20 15:47:08.735920 | PRE-RUN START: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/multinode-customizations.yml@main] 2026-03-20 15:47:09.385379 | 2026-03-20 15:47:09.385527 | PLAY [Tweak CRC node] 2026-03-20 15:47:09.413134 | 2026-03-20 15:47:09.413244 | TASK [Load network parameters] 2026-03-20 15:47:10.045176 | crc | ok 2026-03-20 15:47:10.059801 | 2026-03-20 15:47:10.059958 | TASK [Check which dnsmasq config we must edit] 2026-03-20 15:47:10.512501 | crc | ok 2026-03-20 15:47:10.527470 | 2026-03-20 15:47:10.527615 | TASK [Ensure crc knows about its second NIC] 2026-03-20 15:47:11.604420 | crc | changed 2026-03-20 15:47:11.619659 | 2026-03-20 15:47:11.619829 | TASK [Ensure crc does not get "public" DNS] 2026-03-20 15:47:12.500141 | crc | changed 2026-03-20 15:47:12.516286 | 2026-03-20 15:47:12.516431 | TASK [Restart NetworkManager] 2026-03-20 15:47:13.653705 | crc | changed 2026-03-20 15:47:13.672867 | 2026-03-20 15:47:13.673009 | LOOP [Configure dns forwarders] 2026-03-20 15:47:14.149277 | crc | ok 2026-03-20 15:47:14.435596 | crc | ok 2026-03-20 15:47:14.457973 | 2026-03-20 15:47:14.458113 | TASK [Configure local DNS for CRC pod] 2026-03-20 15:47:14.941493 | crc | ok 2026-03-20 15:47:14.956651 | 2026-03-20 15:47:14.956819 | TASK [Configure dnsmasq listen-address to listen on both br-ex and ci-private-network] 2026-03-20 15:47:15.379304 | crc | changed: line added 2026-03-20 15:47:15.394287 | 2026-03-20 15:47:15.394444 | TASK [Restart native dnsmasq service] 2026-03-20 15:47:15.927277 | crc | changed 2026-03-20 15:47:15.946132 | 2026-03-20 15:47:15.946257 | TASK [Stop dnsmasq] 2026-03-20 15:47:15.974618 | crc | skipping: Conditional result was False 2026-03-20 15:47:15.991847 | 2026-03-20 15:47:15.991988 | TASK [Make sure that crc-dnsmasq is not running] 2026-03-20 15:47:16.019732 | crc | skipping: Conditional result was False 2026-03-20 15:47:16.035585 | 2026-03-20 15:47:16.035760 | TASK [Start dnsmasq] 2026-03-20 15:47:16.063499 | crc | skipping: Conditional result was False 2026-03-20 15:47:16.083576 | 2026-03-20 15:47:16.083745 | TASK [Wait for CRC to be ready] 2026-03-20 15:47:16.713352 | crc | WARNING: Using insecure TLS client config. Setting this option is not supported! 2026-03-20 15:47:16.870818 | crc | 2026-03-20 15:47:16.870918 | crc | Login successful. 2026-03-20 15:47:16.895881 | crc | 2026-03-20 15:47:16.895954 | crc | You have access to 65 projects, the list has been suppressed. You can list all projects with 'oc projects' 2026-03-20 15:47:16.895997 | crc | 2026-03-20 15:47:16.896017 | crc | Using project "default". 2026-03-20 15:47:17.138623 | crc | changed 2026-03-20 15:47:17.155403 | 2026-03-20 15:47:17.155546 | TASK [Set insecure registry on crc node] 2026-03-20 15:47:17.208856 | crc | ok 2026-03-20 15:47:17.231999 | crc | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/untrusted/project_0/github.com/openstack-k8s-operators/ci-framework/ci/playbooks/tasks/set_crc_insecure_registry.yml 2026-03-20 15:47:17.242446 | 2026-03-20 15:47:17.242524 | TASK [Patch the image.config.openshift.io resource to include insecure registry] 2026-03-20 15:47:17.723012 | crc | image.config.openshift.io/cluster patched 2026-03-20 15:47:17.802657 | crc | changed 2026-03-20 15:47:17.817034 | 2026-03-20 15:47:17.817170 | TASK [Patch the image.config.openshift.io resource to allow registries] 2026-03-20 15:47:18.376438 | crc | image.config.openshift.io/cluster patched 2026-03-20 15:47:18.434211 | crc | changed 2026-03-20 15:47:18.448825 | 2026-03-20 15:47:18.448967 | LOOP [Add additional allowed registries] 2026-03-20 15:47:18.502908 | 2026-03-20 15:47:18.503159 | TASK [Ensure registries.conf.d exists] 2026-03-20 15:47:18.905197 | crc | ok 2026-03-20 15:47:18.920274 | 2026-03-20 15:47:18.920417 | TASK [Set Insecure registry for content provider] 2026-03-20 15:47:19.433972 | crc | changed: File created 2026-03-20 15:47:19.449875 | 2026-03-20 15:47:19.450009 | TASK [Set registry mirror override] 2026-03-20 15:47:19.488095 | crc | skipping: Conditional result was False 2026-03-20 15:47:19.504520 | 2026-03-20 15:47:19.504652 | TASK [Restart crio] 2026-03-20 15:47:21.319129 | crc | changed 2026-03-20 15:47:21.370169 | 2026-03-20 15:47:21.370306 | PLAY [Tweak Controller] 2026-03-20 15:47:21.394043 | 2026-03-20 15:47:21.394127 | TASK [Install other packages] 2026-03-20 15:47:27.021475 | controller | changed 2026-03-20 15:47:27.034972 | 2026-03-20 15:47:27.035196 | LOOP [Ensure we know compute host keys] 2026-03-20 15:47:27.413189 | controller | changed 2026-03-20 15:47:27.413434 | 2026-03-20 15:47:27.680469 | controller | changed 2026-03-20 15:47:27.695409 | 2026-03-20 15:47:27.695616 | TASK [Generate an ssh keypair] 2026-03-20 15:47:27.938407 | controller | Generating public/private ecdsa key pair. 2026-03-20 15:47:27.938443 | controller | Your identification has been saved in /home/zuul/.ssh/id_cifw 2026-03-20 15:47:27.938449 | controller | Your public key has been saved in /home/zuul/.ssh/id_cifw.pub 2026-03-20 15:47:27.938454 | controller | The key fingerprint is: 2026-03-20 15:47:27.938459 | controller | SHA256:56BCNfMconmgf04ZKHP4FAZ95HdnF4bQuBJa0xGG0Ts zuul@np0005653376.novalocal 2026-03-20 15:47:27.938463 | controller | The key's randomart image is: 2026-03-20 15:47:27.938468 | controller | +---[ECDSA 256]---+ 2026-03-20 15:47:27.938473 | controller | | .. .. .==* .o | 2026-03-20 15:47:27.938477 | controller | | .... =.+ o. . | 2026-03-20 15:47:27.938481 | controller | | +.*ooo.oo . | 2026-03-20 15:47:27.938485 | controller | | + B.B.oEo . | 2026-03-20 15:47:27.938490 | controller | | = B o S... | 2026-03-20 15:47:27.938494 | controller | | O . + + | 2026-03-20 15:47:27.938498 | controller | | + = . | 2026-03-20 15:47:27.938503 | controller | | = | 2026-03-20 15:47:27.938507 | controller | | . | 2026-03-20 15:47:27.938511 | controller | +----[SHA256]-----+ 2026-03-20 15:47:28.247358 | controller | changed 2026-03-20 15:47:28.263429 | 2026-03-20 15:47:28.263655 | TASK [Get public key] 2026-03-20 15:47:28.516113 | controller | ok 2026-03-20 15:47:28.530401 | 2026-03-20 15:47:28.530533 | TASK [Register pub key as a fact] 2026-03-20 15:47:28.604359 | controller | ok 2026-03-20 15:47:28.616944 | 2026-03-20 15:47:28.617112 | TASK [Load network parameters] 2026-03-20 15:47:28.877139 | controller | ok 2026-03-20 15:47:28.890983 | 2026-03-20 15:47:28.891206 | TASK [Check if cloud-init is overriding NM settings] 2026-03-20 15:47:29.235965 | controller | ok 2026-03-20 15:47:29.244383 | 2026-03-20 15:47:29.244577 | TASK [Remove cloud-init DNS override if present] 2026-03-20 15:47:29.271988 | controller | skipping: Conditional result was False 2026-03-20 15:47:29.280403 | 2026-03-20 15:47:29.280546 | TASK [Reload the NetworkManager to pick the changes] 2026-03-20 15:47:29.306178 | controller | skipping: Conditional result was False 2026-03-20 15:47:29.314341 | 2026-03-20 15:47:29.314412 | TASK [Get the default iface connection] 2026-03-20 15:47:29.641141 | controller | System eth0 2026-03-20 15:47:29.852251 | controller | changed 2026-03-20 15:47:29.868024 | 2026-03-20 15:47:29.868274 | TASK [Prepend CRC DNS server in the controllers default Network Manager connection configuation] 2026-03-20 15:47:30.435780 | controller | Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/5) 2026-03-20 15:47:30.930600 | controller | changed 2026-03-20 15:47:30.979759 | 2026-03-20 15:47:30.979966 | PLAY [Ensure persistent SSH access] 2026-03-20 15:47:31.029207 | 2026-03-20 15:47:31.029317 | TASK [Inject key in ansible_user] 2026-03-20 15:47:31.609432 | controller | changed 2026-03-20 15:47:31.777716 | crc | changed 2026-03-20 15:47:31.790424 | 2026-03-20 15:47:31.790576 | TASK [Inject key in root] 2026-03-20 15:47:32.151291 | controller | changed 2026-03-20 15:47:32.349971 | crc | changed 2026-03-20 15:47:32.425130 | 2026-03-20 15:47:32.425206 | PLAY [Add cloud-admin user on Compute] 2026-03-20 15:47:32.426276 | 2026-03-20 15:47:32.426323 | PLAY [Set some fancy hostname] 2026-03-20 15:47:32.442256 | 2026-03-20 15:47:32.442318 | TASK [Inject hostname in configuration file] 2026-03-20 15:47:33.356480 | controller | changed 2026-03-20 15:47:33.391541 | 2026-03-20 15:47:33.391630 | PLAY RECAP 2026-03-20 15:47:33.391721 | controller | ok: 12 changed: 8 unreachable: 0 failed: 0 skipped: 2 rescued: 0 ignored: 0 2026-03-20 15:47:33.391763 | crc | ok: 18 changed: 12 unreachable: 0 failed: 0 skipped: 5 rescued: 0 ignored: 0 2026-03-20 15:47:33.391788 | 2026-03-20 15:47:33.534892 | PRE-RUN END RESULT_NORMAL: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/multinode-customizations.yml@main] 2026-03-20 15:47:33.543935 | PRE-RUN START: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/e2e-prepare.yml@main] 2026-03-20 15:47:34.181187 | 2026-03-20 15:47:34.181375 | PLAY [Run ci/playbooks/e2e-prepare.yml] 2026-03-20 15:47:34.204046 | 2026-03-20 15:47:34.204120 | TASK [Clone repos in the job workspace] 2026-03-20 15:47:34.226285 | controller | ok 2026-03-20 15:47:34.244616 | 2026-03-20 15:47:34.244711 | TASK [prepare-workspace : Start zuul_console daemon.] 2026-03-20 15:47:34.773413 | controller | ok 2026-03-20 15:47:34.788696 | 2026-03-20 15:47:34.788847 | TASK [prepare-workspace : Synchronize src repos to workspace directory.] 2026-03-20 15:47:42.322921 | controller | Output suppressed because no_log was given 2026-03-20 15:47:42.333528 | 2026-03-20 15:47:42.333593 | TASK [Create zuul-output directory] 2026-03-20 15:47:42.946107 | controller | ok 2026-03-20 15:47:42.959594 | 2026-03-20 15:47:42.959799 | TASK [Install required packages] 2026-03-20 15:47:44.091820 | controller | ok: Nothing to do 2026-03-20 15:47:44.108620 | 2026-03-20 15:47:44.108894 | TASK [Output pip related things] 2026-03-20 15:47:44.771064 | controller | pip 21.3.1 from /usr/lib/python3.9/site-packages/pip (python 3.9) 2026-03-20 15:47:45.171075 | controller | changed 2026-03-20 15:47:45.188510 | 2026-03-20 15:47:45.188692 | TASK [Install requirements] 2026-03-20 15:49:17.234390 | controller | changed 2026-03-20 15:49:17.265006 | 2026-03-20 15:49:17.265175 | TASK [Add cronjob to trigger job stats] 2026-03-20 15:49:17.315786 | controller | skipping: Conditional result was False 2026-03-20 15:49:17.334135 | 2026-03-20 15:49:17.334349 | LOOP [Construct project change list] 2026-03-20 15:49:17.409599 | controller | ok: 2026-03-20 15:49:17.410007 | controller | { 2026-03-20 15:49:17.410065 | controller | "branch": "main", 2026-03-20 15:49:17.410105 | controller | "change": "354", 2026-03-20 15:49:17.410145 | controller | "change_message": "Disable horizon deployment in master release EDPM jobs\n\nHorizon packages in RDO centos10-master is broken and fails to start. Given that we are not running automated horizon tests in the edpm deployment jobs, let's disable it in the jobs deploying from master.\r\n\r\nAssisted-By: Cursor", 2026-03-20 15:49:17.410195 | controller | "change_url": "https://github.com/openstack-k8s-operators/watcher-operator/pull/354", 2026-03-20 15:49:17.410235 | controller | "commit_id": "ee00c2d330b27d46c48ac29a20680b56ca50df3c", 2026-03-20 15:49:17.410271 | controller | "patchset": "ee00c2d330b27d46c48ac29a20680b56ca50df3c", 2026-03-20 15:49:17.410306 | controller | "project": { 2026-03-20 15:49:17.410341 | controller | "canonical_hostname": "github.com", 2026-03-20 15:49:17.410376 | controller | "canonical_name": "github.com/openstack-k8s-operators/watcher-operator", 2026-03-20 15:49:17.410413 | controller | "name": "openstack-k8s-operators/watcher-operator", 2026-03-20 15:49:17.410447 | controller | "short_name": "watcher-operator", 2026-03-20 15:49:17.410482 | controller | "src_dir": "src/github.com/openstack-k8s-operators/watcher-operator" 2026-03-20 15:49:17.410518 | controller | }, 2026-03-20 15:49:17.410552 | controller | "topic": null 2026-03-20 15:49:17.410587 | controller | } 2026-03-20 15:49:17.495683 | 2026-03-20 15:49:17.495832 | PLAY [Run ci/playbooks/build_runner_image.yml] 2026-03-20 15:49:17.533297 | 2026-03-20 15:49:17.533446 | TASK [Filter out host if needed] 2026-03-20 15:49:17.544815 | controller | skipping: end_host conditional evaluated to False, continuing execution for controller 2026-03-20 15:49:17.551522 | 2026-03-20 15:49:17.551579 | TASK [Filter out host if needed] 2026-03-20 15:49:17.559394 | crc | skipping: end_host conditional evaluated to False, continuing execution for crc 2026-03-20 15:49:17.564308 | 2026-03-20 15:49:17.564362 | TASK [Get git tag for image tagging] 2026-03-20 15:49:18.196814 | controller | skipping: Conditional result was False 2026-03-20 15:49:18.197995 | crc | skipping: Conditional result was False 2026-03-20 15:49:18.211551 | 2026-03-20 15:49:18.211733 | TASK [Set openstack-runner image] 2026-03-20 15:49:18.255390 | controller | skipping: Conditional result was False 2026-03-20 15:49:18.277216 | crc | skipping: Conditional result was False 2026-03-20 15:49:18.291101 | 2026-03-20 15:49:18.291235 | TASK [Build openstack-runner image] 2026-03-20 15:49:18.331725 | controller | skipping: Conditional result was False 2026-03-20 15:49:18.353177 | crc | skipping: Conditional result was False 2026-03-20 15:49:18.362896 | 2026-03-20 15:49:18.362985 | TASK [Push openstack-runner image] 2026-03-20 15:49:18.397493 | controller | skipping: Conditional result was False 2026-03-20 15:49:18.418925 | crc | skipping: Conditional result was False 2026-03-20 15:49:18.437730 | 2026-03-20 15:49:18.437888 | TASK [Make sure ci-framework directory exists] 2026-03-20 15:49:18.474828 | controller | skipping: Conditional result was False 2026-03-20 15:49:18.486149 | crc | skipping: Conditional result was False 2026-03-20 15:49:18.497888 | 2026-03-20 15:49:18.497978 | TASK [Create EDPM ansible var file] 2026-03-20 15:49:18.532910 | controller | skipping: Conditional result was False 2026-03-20 15:49:18.554271 | crc | skipping: Conditional result was False 2026-03-20 15:49:18.616811 | 2026-03-20 15:49:18.616966 | PLAY RECAP 2026-03-20 15:49:18.617019 | controller | ok: 7 changed: 3 unreachable: 0 failed: 0 skipped: 7 rescued: 0 ignored: 0 2026-03-20 15:49:18.617047 | crc | ok: 0 changed: 0 unreachable: 0 failed: 0 skipped: 6 rescued: 0 ignored: 0 2026-03-20 15:49:18.617065 | 2026-03-20 15:49:18.779051 | PRE-RUN END RESULT_NORMAL: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/e2e-prepare.yml@main] 2026-03-20 15:49:18.789568 | PRE-RUN START: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/dump_zuul_data.yml@main] 2026-03-20 15:49:19.373270 | 2026-03-20 15:49:19.373411 | PLAY [Run ci/playbooks/dump_zuul_data.yml] 2026-03-20 15:49:19.397408 | 2026-03-20 15:49:19.397487 | TASK [Create zuul-output directory] 2026-03-20 15:49:19.934708 | controller | changed 2026-03-20 15:49:19.947364 | 2026-03-20 15:49:19.947515 | TASK [Slurp Zuul inventory test] 2026-03-20 15:49:20.406246 | controller -> localhost | ok 2026-03-20 15:49:20.421814 | 2026-03-20 15:49:20.421955 | TASK [Save zuul inventory] 2026-03-20 15:49:21.402507 | controller | changed 2026-03-20 15:49:21.415354 | 2026-03-20 15:49:21.415512 | TASK [Save zuul vars without the change_message] 2026-03-20 15:49:22.213054 | controller | changed 2026-03-20 15:49:22.256789 | 2026-03-20 15:49:22.256924 | PLAY RECAP 2026-03-20 15:49:22.257049 | controller | ok: 4 changed: 3 unreachable: 0 failed: 0 skipped: 0 rescued: 0 ignored: 0 2026-03-20 15:49:22.257087 | 2026-03-20 15:49:22.394131 | PRE-RUN END RESULT_NORMAL: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/dump_zuul_data.yml@main] 2026-03-20 15:49:22.402955 | PRE-RUN START: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/kuttl/kuttl-from-operator-deps.yaml@main] 2026-03-20 15:49:22.997648 | 2026-03-20 15:49:22.997814 | PLAY [Run ci/playbooks/kuttl/deploy-deps.yml] 2026-03-20 15:49:23.021307 | 2026-03-20 15:49:23.021391 | TASK [Run kuttl deploy-deps playbook] 2026-03-20 15:49:24.440772 | controller | 2026-03-20 15:49:25.680234 | controller | PLAY [Bootstrap playbook] ****************************************************** 2026-03-20 15:49:25.680313 | controller | 2026-03-20 15:49:25.680322 | controller | TASK [Gathering Facts ] ******************************************************** 2026-03-20 15:49:25.680329 | controller | Friday 20 March 2026 15:49:24 +0000 (0:00:00.064) 0:00:00.064 ********** 2026-03-20 15:49:25.680334 | controller | Friday 20 March 2026 15:49:24 +0000 (0:00:00.063) 0:00:00.063 ********** 2026-03-20 15:49:25.680346 | controller | ok: [controller] 2026-03-20 15:49:25.713313 | controller | 2026-03-20 15:49:25.713356 | controller | TASK [cifmw_setup : Set custom cifmw PATH reusable fact cifmw_path={{ ansible_user_dir }}/.crc/bin:{{ ansible_user_dir }}/.crc/bin/oc:{{ ansible_user_dir }}/bin:{{ ansible_env.PATH }}, cacheable=True] *** 2026-03-20 15:49:25.713365 | controller | Friday 20 March 2026 15:49:25 +0000 (0:00:01.239) 0:00:01.304 ********** 2026-03-20 15:49:25.713370 | controller | Friday 20 March 2026 15:49:25 +0000 (0:00:01.239) 0:00:01.303 ********** 2026-03-20 15:49:25.713380 | controller | ok: [controller] 2026-03-20 15:49:25.796046 | controller | 2026-03-20 15:49:25.796083 | controller | TASK [cifmw_setup : Get customized parameters ci_framework_params={{ 2026-03-20 15:49:25.796090 | controller | hostvars[inventory_hostname] | 2026-03-20 15:49:25.796099 | controller | dict2items | 2026-03-20 15:49:25.796104 | controller | selectattr("key", "match", 2026-03-20 15:49:25.796109 | controller | "^(cifmw|pre|post)_(?!install_yamls|openshift_token|openshift_login|openshift_kubeconfig).*") | 2026-03-20 15:49:25.796114 | controller | list | items2dict 2026-03-20 15:49:25.796118 | controller | }}] *** 2026-03-20 15:49:25.796123 | controller | Friday 20 March 2026 15:49:25 +0000 (0:00:00.033) 0:00:01.337 ********** 2026-03-20 15:49:25.796128 | controller | Friday 20 March 2026 15:49:25 +0000 (0:00:00.033) 0:00:01.336 ********** 2026-03-20 15:49:25.796137 | controller | ok: [controller] 2026-03-20 15:49:26.139746 | controller | 2026-03-20 15:49:26.139821 | controller | TASK [install_ca : Ensure target directory exists path={{ cifmw_install_ca_trust_dir }}, state=directory, mode=0755] *** 2026-03-20 15:49:26.139828 | controller | Friday 20 March 2026 15:49:25 +0000 (0:00:00.082) 0:00:01.420 ********** 2026-03-20 15:49:26.139833 | controller | Friday 20 March 2026 15:49:25 +0000 (0:00:00.082) 0:00:01.419 ********** 2026-03-20 15:49:26.139848 | controller | ok: [controller] 2026-03-20 15:49:26.160605 | controller | 2026-03-20 15:49:26.160664 | controller | TASK [install_ca : Install internal CA from url url={{ cifmw_install_ca_url }}, dest={{ cifmw_install_ca_trust_dir }}, validate_certs={{ cifmw_install_ca_url_validate_certs | default(omit) }}, mode=0644] *** 2026-03-20 15:49:26.160671 | controller | Friday 20 March 2026 15:49:26 +0000 (0:00:00.343) 0:00:01.763 ********** 2026-03-20 15:49:26.160676 | controller | Friday 20 March 2026 15:49:26 +0000 (0:00:00.343) 0:00:01.762 ********** 2026-03-20 15:49:26.160708 | controller | skipping: [controller] 2026-03-20 15:49:26.184467 | controller | 2026-03-20 15:49:26.184530 | controller | TASK [install_ca : Install custom CA bundle from inline dest={{ cifmw_install_ca_trust_dir }}/cifmw_inline_ca_bundle.crt, content={{ cifmw_install_ca_bundle_inline }}, mode=0644] *** 2026-03-20 15:49:26.184536 | controller | Friday 20 March 2026 15:49:26 +0000 (0:00:00.021) 0:00:01.784 ********** 2026-03-20 15:49:26.184541 | controller | Friday 20 March 2026 15:49:26 +0000 (0:00:00.021) 0:00:01.783 ********** 2026-03-20 15:49:26.184555 | controller | skipping: [controller] 2026-03-20 15:49:26.210894 | controller | 2026-03-20 15:49:26.210958 | controller | TASK [install_ca : Install custom CA bundle from file dest={{ cifmw_install_ca_trust_dir }}/{{ cifmw_install_ca_bundle_src | basename }}, src={{ cifmw_install_ca_bundle_src }}, mode=0644] *** 2026-03-20 15:49:26.210964 | controller | Friday 20 March 2026 15:49:26 +0000 (0:00:00.023) 0:00:01.808 ********** 2026-03-20 15:49:26.210969 | controller | Friday 20 March 2026 15:49:26 +0000 (0:00:00.023) 0:00:01.807 ********** 2026-03-20 15:49:26.210984 | controller | skipping: [controller] 2026-03-20 15:49:26.241246 | controller | 2026-03-20 15:49:26.241298 | controller | TASK [install_ca : Update ca bundle _raw_params=update-ca-trust] *************** 2026-03-20 15:49:26.241311 | controller | Friday 20 March 2026 15:49:26 +0000 (0:00:00.026) 0:00:01.834 ********** 2026-03-20 15:49:26.241327 | controller | Friday 20 March 2026 15:49:26 +0000 (0:00:00.026) 0:00:01.833 ********** 2026-03-20 15:49:26.241342 | controller | skipping: [controller] 2026-03-20 15:49:26.818855 | controller | 2026-03-20 15:49:26.818900 | controller | TASK [repo_setup : Ensure directories are present path={{ cifmw_repo_setup_basedir }}/{{ item }}, state=directory, mode=0755] *** 2026-03-20 15:49:26.818908 | controller | Friday 20 March 2026 15:49:26 +0000 (0:00:00.030) 0:00:01.865 ********** 2026-03-20 15:49:26.818915 | controller | Friday 20 March 2026 15:49:26 +0000 (0:00:00.030) 0:00:01.864 ********** 2026-03-20 15:49:26.818928 | controller | changed: [controller] => (item=tmp) 2026-03-20 15:49:27.817628 | controller | changed: [controller] => (item=artifacts/repositories) 2026-03-20 15:49:27.817658 | controller | changed: [controller] => (item=venv/repo_setup) 2026-03-20 15:49:27.817666 | controller | 2026-03-20 15:49:27.817671 | controller | TASK [repo_setup : Make sure git-core package is installed name=git-core, state=present] *** 2026-03-20 15:49:27.817675 | controller | Friday 20 March 2026 15:49:26 +0000 (0:00:00.577) 0:00:02.443 ********** 2026-03-20 15:49:27.817680 | controller | Friday 20 March 2026 15:49:26 +0000 (0:00:00.577) 0:00:02.441 ********** 2026-03-20 15:49:27.817708 | controller | ok: [controller] 2026-03-20 15:49:28.926175 | controller | 2026-03-20 15:49:28.926215 | controller | TASK [repo_setup : Get repo-setup repository accept_hostkey=True, dest={{ cifmw_repo_setup_basedir }}/tmp/repo-setup, repo={{ cifmw_repo_setup_src }}] *** 2026-03-20 15:49:28.926224 | controller | Friday 20 March 2026 15:49:27 +0000 (0:00:00.998) 0:00:03.441 ********** 2026-03-20 15:49:28.926231 | controller | Friday 20 March 2026 15:49:27 +0000 (0:00:00.998) 0:00:03.440 ********** 2026-03-20 15:49:28.926250 | controller | changed: [controller] 2026-03-20 15:49:37.976648 | controller | 2026-03-20 15:49:37.976759 | controller | TASK [repo_setup : Initialize python venv and install requirements virtualenv={{ cifmw_repo_setup_venv }}, requirements={{ cifmw_repo_setup_basedir }}/tmp/repo-setup/requirements.txt, virtualenv_command=python3 -m venv --system-site-packages --upgrade-deps] *** 2026-03-20 15:49:37.976773 | controller | Friday 20 March 2026 15:49:28 +0000 (0:00:01.108) 0:00:04.550 ********** 2026-03-20 15:49:37.976779 | controller | Friday 20 March 2026 15:49:28 +0000 (0:00:01.108) 0:00:04.549 ********** 2026-03-20 15:49:37.976799 | controller | changed: [controller] 2026-03-20 15:49:38.936222 | controller | 2026-03-20 15:49:38.936272 | controller | TASK [repo_setup : Install repo-setup package chdir={{ cifmw_repo_setup_basedir }}/tmp/repo-setup, creates={{ cifmw_repo_setup_venv }}/bin/repo-setup, _raw_params={{ cifmw_repo_setup_venv }}/bin/python setup.py install] *** 2026-03-20 15:49:38.936288 | controller | Friday 20 March 2026 15:49:37 +0000 (0:00:09.050) 0:00:13.600 ********** 2026-03-20 15:49:38.936299 | controller | Friday 20 March 2026 15:49:37 +0000 (0:00:09.050) 0:00:13.599 ********** 2026-03-20 15:49:38.936317 | controller | changed: [controller] 2026-03-20 15:49:38.958000 | controller | 2026-03-20 15:49:38.958031 | controller | TASK [repo_setup : Set cifmw_repo_setup_dlrn_hash_tag from content provider cifmw_repo_setup_dlrn_hash_tag={{ content_provider_dlrn_md5_hash }}] *** 2026-03-20 15:49:38.958038 | controller | Friday 20 March 2026 15:49:38 +0000 (0:00:00.959) 0:00:14.559 ********** 2026-03-20 15:49:38.958043 | controller | Friday 20 March 2026 15:49:38 +0000 (0:00:00.959) 0:00:14.558 ********** 2026-03-20 15:49:38.958052 | controller | skipping: [controller] 2026-03-20 15:49:39.585772 | controller | 2026-03-20 15:49:39.585877 | controller | TASK [repo_setup : Run repo-setup _raw_params={{ cifmw_repo_setup_venv }}/bin/repo-setup {{ cifmw_repo_setup_promotion }} {{ cifmw_repo_setup_additional_repos }} -d {{ cifmw_repo_setup_os_release }}{{ cifmw_repo_setup_dist_major_version }} -b {{ cifmw_repo_setup_branch }} --rdo-mirror {{ cifmw_repo_setup_rdo_mirror }} {% if cifmw_repo_setup_dlrn_hash_tag | length > 0 %} --dlrn-hash-tag {{ cifmw_repo_setup_dlrn_hash_tag }} {% endif %} -o {{ cifmw_repo_setup_output }}] *** 2026-03-20 15:49:39.585886 | controller | Friday 20 March 2026 15:49:38 +0000 (0:00:00.022) 0:00:14.582 ********** 2026-03-20 15:49:39.585892 | controller | Friday 20 March 2026 15:49:38 +0000 (0:00:00.022) 0:00:14.581 ********** 2026-03-20 15:49:39.585919 | controller | changed: [controller] 2026-03-20 15:49:39.620063 | controller | 2026-03-20 15:49:39.620110 | controller | TASK [repo_setup : Get component repo url={{ cifmw_repo_setup_dlrn_uri }}/{{ cifmw_repo_setup_os_release }}{{ cifmw_repo_setup_dist_major_version }}-{{ cifmw_repo_setup_branch }}/component/{{ cifmw_repo_setup_component_name }}/{{ cifmw_repo_setup_component_promotion_tag }}/delorean.repo, dest={{ cifmw_repo_setup_output }}/{{ cifmw_repo_setup_component_name }}_{{ cifmw_repo_setup_component_promotion_tag }}_delorean.repo, mode=0644] *** 2026-03-20 15:49:39.620117 | controller | Friday 20 March 2026 15:49:39 +0000 (0:00:00.627) 0:00:15.209 ********** 2026-03-20 15:49:39.620122 | controller | Friday 20 March 2026 15:49:39 +0000 (0:00:00.627) 0:00:15.208 ********** 2026-03-20 15:49:39.620132 | controller | skipping: [controller] 2026-03-20 15:49:39.650597 | controller | 2026-03-20 15:49:39.650651 | controller | TASK [repo_setup : Rename component repo path={{ cifmw_repo_setup_output }}/{{ cifmw_repo_setup_component_name }}_{{ cifmw_repo_setup_component_promotion_tag }}_delorean.repo, regexp=delorean-component-{{ cifmw_repo_setup_component_name }}, replace={{ cifmw_repo_setup_component_name }}-{{ cifmw_repo_setup_component_promotion_tag }}] *** 2026-03-20 15:49:39.650659 | controller | Friday 20 March 2026 15:49:39 +0000 (0:00:00.034) 0:00:15.244 ********** 2026-03-20 15:49:39.650664 | controller | Friday 20 March 2026 15:49:39 +0000 (0:00:00.034) 0:00:15.242 ********** 2026-03-20 15:49:39.650674 | controller | skipping: [controller] 2026-03-20 15:49:39.684160 | controller | 2026-03-20 15:49:39.684208 | controller | TASK [repo_setup : Disable component repo in current-podified dlrn repo path={{ cifmw_repo_setup_output }}/delorean.repo, section=delorean-component-{{ cifmw_repo_setup_component_name }}, option=enabled, value=0, mode=0644] *** 2026-03-20 15:49:39.684214 | controller | Friday 20 March 2026 15:49:39 +0000 (0:00:00.030) 0:00:15.274 ********** 2026-03-20 15:49:39.684219 | controller | Friday 20 March 2026 15:49:39 +0000 (0:00:00.030) 0:00:15.273 ********** 2026-03-20 15:49:39.684229 | controller | skipping: [controller] 2026-03-20 15:49:40.149752 | controller | 2026-03-20 15:49:40.149795 | controller | TASK [repo_setup : Run repo-setup-get-hash _raw_params={{ cifmw_repo_setup_venv }}/bin/repo-setup-get-hash --dlrn-url {{ cifmw_repo_setup_dlrn_uri[:-1] }} --os-version {{ cifmw_repo_setup_os_release }}{{ cifmw_repo_setup_dist_major_version }} --release {{ cifmw_repo_setup_branch }} {% if cifmw_repo_setup_component_name | length > 0 -%} --component {{ cifmw_repo_setup_component_name }} --tag {{ cifmw_repo_setup_component_promotion_tag }} {% else -%} --tag {{cifmw_repo_setup_promotion }} {% endif -%} {% if (cifmw_repo_setup_dlrn_hash_tag | length > 0) and (cifmw_repo_setup_component_name | length <= 0) -%} --dlrn-hash-tag {{ cifmw_repo_setup_dlrn_hash_tag }} {% endif -%} --json] *** 2026-03-20 15:49:40.149805 | controller | Friday 20 March 2026 15:49:39 +0000 (0:00:00.033) 0:00:15.308 ********** 2026-03-20 15:49:40.149815 | controller | Friday 20 March 2026 15:49:39 +0000 (0:00:00.033) 0:00:15.307 ********** 2026-03-20 15:49:40.149828 | controller | changed: [controller] 2026-03-20 15:49:40.964514 | controller | 2026-03-20 15:49:40.964573 | controller | TASK [repo_setup : Dump full hash in delorean.repo.md5 file content={{ _repo_setup_json['full_hash'] }} 2026-03-20 15:49:40.964579 | controller | , dest={{ cifmw_repo_setup_basedir }}/artifacts/repositories/delorean.repo.md5, mode=0644] *** 2026-03-20 15:49:40.964592 | controller | Friday 20 March 2026 15:49:40 +0000 (0:00:00.465) 0:00:15.773 ********** 2026-03-20 15:49:40.964597 | controller | Friday 20 March 2026 15:49:40 +0000 (0:00:00.465) 0:00:15.772 ********** 2026-03-20 15:49:40.964609 | controller | changed: [controller] 2026-03-20 15:49:40.983713 | controller | 2026-03-20 15:49:40.983783 | controller | TASK [repo_setup : Dump current-podified hash url={{ cifmw_repo_setup_dlrn_uri }}/{{ cifmw_repo_setup_os_release }}{{ cifmw_repo_setup_dist_major_version }}-{{ cifmw_repo_setup_branch }}/current-podified/delorean.repo.md5, dest={{ cifmw_repo_setup_basedir }}/artifacts/repositories/delorean.repo.md5, mode=0644] *** 2026-03-20 15:49:40.983795 | controller | Friday 20 March 2026 15:49:40 +0000 (0:00:00.814) 0:00:16.588 ********** 2026-03-20 15:49:40.983808 | controller | Friday 20 March 2026 15:49:40 +0000 (0:00:00.814) 0:00:16.587 ********** 2026-03-20 15:49:40.983819 | controller | skipping: [controller] 2026-03-20 15:49:41.004218 | controller | 2026-03-20 15:49:41.004264 | controller | TASK [repo_setup : Slurp current podified hash src={{ cifmw_repo_setup_basedir }}/artifacts/repositories/delorean.repo.md5] *** 2026-03-20 15:49:41.004270 | controller | Friday 20 March 2026 15:49:40 +0000 (0:00:00.019) 0:00:16.607 ********** 2026-03-20 15:49:41.004276 | controller | Friday 20 March 2026 15:49:40 +0000 (0:00:00.019) 0:00:16.606 ********** 2026-03-20 15:49:41.004286 | controller | skipping: [controller] 2026-03-20 15:49:41.024765 | controller | 2026-03-20 15:49:41.024811 | controller | TASK [repo_setup : Update the value of full_hash _repo_setup_json={{ _repo_setup_json | combine({'full_hash': _hash}, recursive=true) }}] *** 2026-03-20 15:49:41.024818 | controller | Friday 20 March 2026 15:49:41 +0000 (0:00:00.020) 0:00:16.628 ********** 2026-03-20 15:49:41.024824 | controller | Friday 20 March 2026 15:49:41 +0000 (0:00:00.020) 0:00:16.627 ********** 2026-03-20 15:49:41.024834 | controller | skipping: [controller] 2026-03-20 15:49:41.055157 | controller | 2026-03-20 15:49:41.055206 | controller | TASK [repo_setup : Export hashes facts for further use cifmw_repo_setup_full_hash={{ _repo_setup_json['full_hash'] }}, cifmw_repo_setup_commit_hash={{ _repo_setup_json['commit_hash'] }}, cifmw_repo_setup_distro_hash={{ _repo_setup_json['distro_hash'] }}, cifmw_repo_setup_extended_hash={{ _repo_setup_json['extended_hash'] }}, cifmw_repo_setup_dlrn_api_url={{ _repo_setup_json['dlrn_api_url'] }}, cifmw_repo_setup_dlrn_url={{ _repo_setup_json['dlrn_url'] }}, cifmw_repo_setup_release={{ _repo_setup_json['release'] }}, cacheable=True] *** 2026-03-20 15:49:41.055217 | controller | Friday 20 March 2026 15:49:41 +0000 (0:00:00.020) 0:00:16.648 ********** 2026-03-20 15:49:41.055224 | controller | Friday 20 March 2026 15:49:41 +0000 (0:00:00.020) 0:00:16.647 ********** 2026-03-20 15:49:41.055237 | controller | ok: [controller] 2026-03-20 15:49:41.073057 | controller | 2026-03-20 15:49:41.073098 | controller | TASK [repo_setup : Create download directory path={{ cifmw_repo_setup_rhos_release_path }}, state=directory, mode=0755] *** 2026-03-20 15:49:41.073107 | controller | Friday 20 March 2026 15:49:41 +0000 (0:00:00.030) 0:00:16.679 ********** 2026-03-20 15:49:41.073114 | controller | Friday 20 March 2026 15:49:41 +0000 (0:00:00.030) 0:00:16.678 ********** 2026-03-20 15:49:41.073126 | controller | skipping: [controller] 2026-03-20 15:49:41.093122 | controller | 2026-03-20 15:49:41.093158 | controller | TASK [repo_setup : Print the URL to request msg={{ cifmw_repo_setup_rhos_release_rpm }}] *** 2026-03-20 15:49:41.093164 | controller | Friday 20 March 2026 15:49:41 +0000 (0:00:00.017) 0:00:16.697 ********** 2026-03-20 15:49:41.093169 | controller | Friday 20 March 2026 15:49:41 +0000 (0:00:00.017) 0:00:16.696 ********** 2026-03-20 15:49:41.093179 | controller | skipping: [controller] 2026-03-20 15:49:41.115573 | controller | 2026-03-20 15:49:41.115619 | controller | TASK [Download the RPM name=krb_request] *************************************** 2026-03-20 15:49:41.115628 | controller | Friday 20 March 2026 15:49:41 +0000 (0:00:00.020) 0:00:16.717 ********** 2026-03-20 15:49:41.115635 | controller | Friday 20 March 2026 15:49:41 +0000 (0:00:00.020) 0:00:16.716 ********** 2026-03-20 15:49:41.115649 | controller | skipping: [controller] 2026-03-20 15:49:41.135474 | controller | 2026-03-20 15:49:41.135537 | controller | TASK [repo_setup : Install RHOS Release tool name={{ 2026-03-20 15:49:41.135547 | controller | cifmw_repo_setup_rhos_release_rpm 2026-03-20 15:49:41.135554 | controller | if cifmw_repo_setup_rhos_release_rpm is not url 2026-03-20 15:49:41.135561 | controller | else cifmw_krb_request_out.path 2026-03-20 15:49:41.135567 | controller | }}, state=present, disable_gpg_check={{ cifmw_repo_setup_rhos_release_gpg_check | bool }}] *** 2026-03-20 15:49:41.135573 | controller | Friday 20 March 2026 15:49:41 +0000 (0:00:00.022) 0:00:16.739 ********** 2026-03-20 15:49:41.135579 | controller | Friday 20 March 2026 15:49:41 +0000 (0:00:00.022) 0:00:16.738 ********** 2026-03-20 15:49:41.135593 | controller | skipping: [controller] 2026-03-20 15:49:41.154466 | controller | 2026-03-20 15:49:41.154674 | controller | TASK [repo_setup : Get rhos-release tool version _raw_params=rhos-release --version] *** 2026-03-20 15:49:41.154716 | controller | Friday 20 March 2026 15:49:41 +0000 (0:00:00.019) 0:00:16.759 ********** 2026-03-20 15:49:41.154727 | controller | Friday 20 March 2026 15:49:41 +0000 (0:00:00.019) 0:00:16.758 ********** 2026-03-20 15:49:41.154740 | controller | skipping: [controller] 2026-03-20 15:49:41.177487 | controller | 2026-03-20 15:49:41.177523 | controller | TASK [repo_setup : Print rhos-release tool version msg={{ rr_version.stdout }}] *** 2026-03-20 15:49:41.177535 | controller | Friday 20 March 2026 15:49:41 +0000 (0:00:00.019) 0:00:16.778 ********** 2026-03-20 15:49:41.177540 | controller | Friday 20 March 2026 15:49:41 +0000 (0:00:00.019) 0:00:16.777 ********** 2026-03-20 15:49:41.177550 | controller | skipping: [controller] 2026-03-20 15:49:41.192425 | controller | 2026-03-20 15:49:41.192457 | controller | TASK [repo_setup : Generate repos using rhos-release {{ cifmw_repo_setup_rhos_release_args }} _raw_params=rhos-release {{ cifmw_repo_setup_rhos_release_args }} \ 2026-03-20 15:49:41.192464 | controller | -t {{ cifmw_repo_setup_output }}] *** 2026-03-20 15:49:41.192469 | controller | Friday 20 March 2026 15:49:41 +0000 (0:00:00.022) 0:00:16.801 ********** 2026-03-20 15:49:41.192474 | controller | Friday 20 March 2026 15:49:41 +0000 (0:00:00.022) 0:00:16.800 ********** 2026-03-20 15:49:41.192483 | controller | skipping: [controller] 2026-03-20 15:49:41.430954 | controller | 2026-03-20 15:49:41.431025 | controller | TASK [repo_setup : Check for /etc/ci/mirror_info.sh path=/etc/ci/mirror_info.sh] *** 2026-03-20 15:49:41.431035 | controller | Friday 20 March 2026 15:49:41 +0000 (0:00:00.015) 0:00:16.816 ********** 2026-03-20 15:49:41.431041 | controller | Friday 20 March 2026 15:49:41 +0000 (0:00:00.015) 0:00:16.815 ********** 2026-03-20 15:49:41.431060 | controller | ok: [controller] 2026-03-20 15:49:41.657587 | controller | 2026-03-20 15:49:41.657654 | controller | TASK [repo_setup : Use RDO proxy mirrors chdir={{ cifmw_repo_setup_output }}, _raw_params=set -o pipefail 2026-03-20 15:49:41.657661 | controller | source /etc/ci/mirror_info.sh 2026-03-20 15:49:41.657669 | controller | sed -i -e "s|https://trunk.rdoproject.org|$NODEPOOL_RDO_PROXY|g" *.repo 2026-03-20 15:49:41.657675 | controller | ] *** 2026-03-20 15:49:41.657681 | controller | Friday 20 March 2026 15:49:41 +0000 (0:00:00.238) 0:00:17.055 ********** 2026-03-20 15:49:41.657701 | controller | Friday 20 March 2026 15:49:41 +0000 (0:00:00.238) 0:00:17.053 ********** 2026-03-20 15:49:41.657720 | controller | changed: [controller] 2026-03-20 15:49:41.905086 | controller | 2026-03-20 15:49:41.905138 | controller | TASK [repo_setup : Use RDO CentOS mirrors (remove CentOS 10 conditional when Nodepool mirrors exist) chdir={{ cifmw_repo_setup_output }}, _raw_params=set -o pipefail 2026-03-20 15:49:41.905144 | controller | source /etc/ci/mirror_info.sh 2026-03-20 15:49:41.905150 | controller | sed -i -e "s|http://mirror.stream.centos.org|$NODEPOOL_CENTOS_MIRROR|g" *.repo 2026-03-20 15:49:41.905155 | controller | ] *** 2026-03-20 15:49:41.905160 | controller | Friday 20 March 2026 15:49:41 +0000 (0:00:00.226) 0:00:17.281 ********** 2026-03-20 15:49:41.905164 | controller | Friday 20 March 2026 15:49:41 +0000 (0:00:00.226) 0:00:17.280 ********** 2026-03-20 15:49:41.905175 | controller | changed: [controller] 2026-03-20 15:49:42.504546 | controller | 2026-03-20 15:49:42.504610 | controller | TASK [repo_setup : Check for gating.repo file on content provider url=http://{{ content_provider_registry_ip }}:8766/gating.repo] *** 2026-03-20 15:49:42.504617 | controller | Friday 20 March 2026 15:49:41 +0000 (0:00:00.247) 0:00:17.529 ********** 2026-03-20 15:49:42.504623 | controller | Friday 20 March 2026 15:49:41 +0000 (0:00:00.247) 0:00:17.527 ********** 2026-03-20 15:49:42.504633 | controller | task path: /home/zuul/src/github.com/openstack-k8s-operators/ci-framework/roles/repo_setup/tasks/populate_gating_repo.yml:2 2026-03-20 15:49:42.540861 | controller | fatal: [controller]: FAILED! => 2026-03-20 15:49:42.540897 | controller | changed: false 2026-03-20 15:49:42.540903 | controller | elapsed: 0 2026-03-20 15:49:42.540910 | controller | msg: 'Status code was -1 and not [200]: Request failed: ' 2026-03-20 15:49:42.540922 | controller | redirected: false 2026-03-20 15:49:42.540928 | controller | status: -1 2026-03-20 15:49:42.540937 | controller | url: http://38.102.83.147:8766/gating.repo 2026-03-20 15:49:42.540951 | controller | ...ignoring 2026-03-20 15:49:42.540958 | controller | 2026-03-20 15:49:42.540964 | controller | TASK [repo_setup : Populate gating repo from content provider ip content=[gating-repo] 2026-03-20 15:49:42.540970 | controller | baseurl=http://{{ content_provider_registry_ip }}:8766/ 2026-03-20 15:49:42.540975 | controller | enabled=1 2026-03-20 15:49:42.540981 | controller | gpgcheck=0 2026-03-20 15:49:42.540987 | controller | priority=1 2026-03-20 15:49:42.540993 | controller | , dest={{ cifmw_repo_setup_output }}/gating.repo, mode=0644] *** 2026-03-20 15:49:42.540999 | controller | Friday 20 March 2026 15:49:42 +0000 (0:00:00.599) 0:00:18.128 ********** 2026-03-20 15:49:42.541005 | controller | Friday 20 March 2026 15:49:42 +0000 (0:00:00.599) 0:00:18.127 ********** 2026-03-20 15:49:42.541016 | controller | skipping: [controller] 2026-03-20 15:49:42.580166 | controller | 2026-03-20 15:49:42.580199 | controller | TASK [repo_setup : Check for DLRN repo at the destination path={{ cifmw_repo_setup_output }}/delorean.repo] *** 2026-03-20 15:49:42.580207 | controller | Friday 20 March 2026 15:49:42 +0000 (0:00:00.036) 0:00:18.165 ********** 2026-03-20 15:49:42.580212 | controller | Friday 20 March 2026 15:49:42 +0000 (0:00:00.036) 0:00:18.163 ********** 2026-03-20 15:49:42.580221 | controller | skipping: [controller] 2026-03-20 15:49:42.615990 | controller | 2026-03-20 15:49:42.616031 | controller | TASK [repo_setup : Lower the priority of DLRN repos to allow installation from gating repo path={{ cifmw_repo_setup_output }}/delorean.repo, regexp=priority=1, replace=priority=20] *** 2026-03-20 15:49:42.616039 | controller | Friday 20 March 2026 15:49:42 +0000 (0:00:00.039) 0:00:18.204 ********** 2026-03-20 15:49:42.616046 | controller | Friday 20 March 2026 15:49:42 +0000 (0:00:00.039) 0:00:18.203 ********** 2026-03-20 15:49:42.616058 | controller | skipping: [controller] 2026-03-20 15:49:42.657789 | controller | 2026-03-20 15:49:42.657829 | controller | TASK [repo_setup : Check for DLRN component repo path={{ cifmw_repo_setup_output }}/{{ _comp_repo }}] *** 2026-03-20 15:49:42.657837 | controller | Friday 20 March 2026 15:49:42 +0000 (0:00:00.035) 0:00:18.240 ********** 2026-03-20 15:49:42.657843 | controller | Friday 20 March 2026 15:49:42 +0000 (0:00:00.035) 0:00:18.238 ********** 2026-03-20 15:49:42.657855 | controller | skipping: [controller] 2026-03-20 15:49:42.693457 | controller | 2026-03-20 15:49:42.693492 | controller | TASK [repo_setup : Lower the priority of component repos to allow installation from gating repo path={{ cifmw_repo_setup_output }}/{{ _comp_repo }}, regexp=priority=1, replace=priority=2] *** 2026-03-20 15:49:42.693500 | controller | Friday 20 March 2026 15:49:42 +0000 (0:00:00.041) 0:00:18.281 ********** 2026-03-20 15:49:42.693507 | controller | Friday 20 March 2026 15:49:42 +0000 (0:00:00.041) 0:00:18.280 ********** 2026-03-20 15:49:42.693518 | controller | skipping: [controller] 2026-03-20 15:49:43.052290 | controller | 2026-03-20 15:49:43.052340 | controller | TASK [repo_setup : Find existing repos from /etc/yum.repos.d directory paths=/etc/yum.repos.d/, patterns=*.repo, recurse=False] *** 2026-03-20 15:49:43.052347 | controller | Friday 20 March 2026 15:49:42 +0000 (0:00:00.035) 0:00:18.316 ********** 2026-03-20 15:49:43.052352 | controller | Friday 20 March 2026 15:49:42 +0000 (0:00:00.035) 0:00:18.315 ********** 2026-03-20 15:49:43.052364 | controller | ok: [controller] 2026-03-20 15:49:43.518045 | controller | 2026-03-20 15:49:43.518102 | controller | TASK [repo_setup : Remove existing repos from /etc/yum.repos.d directory path={{ item }}, state=absent] *** 2026-03-20 15:49:43.518118 | controller | Friday 20 March 2026 15:49:43 +0000 (0:00:00.359) 0:00:18.676 ********** 2026-03-20 15:49:43.518132 | controller | Friday 20 March 2026 15:49:43 +0000 (0:00:00.359) 0:00:18.675 ********** 2026-03-20 15:49:43.518153 | controller | changed: [controller] => (item=/etc/yum.repos.d/centos-addons.repo) 2026-03-20 15:49:43.955704 | controller | changed: [controller] => (item=/etc/yum.repos.d/centos.repo) 2026-03-20 15:49:43.955759 | controller | 2026-03-20 15:49:43.955766 | controller | TASK [repo_setup : Cleanup existing metadata _raw_params=dnf clean metadata] *** 2026-03-20 15:49:43.955771 | controller | Friday 20 March 2026 15:49:43 +0000 (0:00:00.465) 0:00:19.141 ********** 2026-03-20 15:49:43.955783 | controller | Friday 20 March 2026 15:49:43 +0000 (0:00:00.465) 0:00:19.140 ********** 2026-03-20 15:49:43.955795 | controller | changed: [controller] 2026-03-20 15:49:44.288613 | controller | 2026-03-20 15:49:44.288644 | controller | TASK [repo_setup : Copy generated repos to /etc/yum.repos.d directory mode=0755, remote_src=True, src={{ cifmw_repo_setup_output }}/, dest=/etc/yum.repos.d] *** 2026-03-20 15:49:44.288650 | controller | Friday 20 March 2026 15:49:43 +0000 (0:00:00.438) 0:00:19.579 ********** 2026-03-20 15:49:44.288656 | controller | Friday 20 March 2026 15:49:43 +0000 (0:00:00.438) 0:00:19.578 ********** 2026-03-20 15:49:44.288665 | controller | changed: [controller] 2026-03-20 15:49:44.339258 | controller | 2026-03-20 15:49:44.339295 | controller | TASK [ci_setup : Gather variables for each operating system _raw_params={{ item }}] *** 2026-03-20 15:49:44.339305 | controller | Friday 20 March 2026 15:49:44 +0000 (0:00:00.333) 0:00:19.912 ********** 2026-03-20 15:49:44.339312 | controller | Friday 20 March 2026 15:49:44 +0000 (0:00:00.333) 0:00:19.911 ********** 2026-03-20 15:49:44.339324 | controller | ok: [controller] => (item=/home/zuul/src/github.com/openstack-k8s-operators/ci-framework/roles/ci_setup/vars/redhat.yml) 2026-03-20 15:49:44.364991 | controller | 2026-03-20 15:49:44.365030 | controller | TASK [ci_setup : List packages to install var=cifmw_ci_setup_packages] ********* 2026-03-20 15:49:44.365042 | controller | Friday 20 March 2026 15:49:44 +0000 (0:00:00.050) 0:00:19.963 ********** 2026-03-20 15:49:44.365052 | controller | Friday 20 March 2026 15:49:44 +0000 (0:00:00.050) 0:00:19.962 ********** 2026-03-20 15:49:44.365068 | controller | ok: [controller] => 2026-03-20 15:50:17.276221 | controller | cifmw_ci_setup_packages: 2026-03-20 15:50:17.276272 | controller | - bash-completion 2026-03-20 15:50:17.276280 | controller | - ca-certificates 2026-03-20 15:50:17.276286 | controller | - git-core 2026-03-20 15:50:17.276292 | controller | - make 2026-03-20 15:50:17.276299 | controller | - tar 2026-03-20 15:50:17.276305 | controller | - tmux 2026-03-20 15:50:17.276312 | controller | - python3-pip 2026-03-20 15:50:17.276318 | controller | 2026-03-20 15:50:17.276325 | controller | TASK [ci_setup : Install needed packages name={{ cifmw_ci_setup_packages }}, state=latest] *** 2026-03-20 15:50:17.276332 | controller | Friday 20 March 2026 15:49:44 +0000 (0:00:00.025) 0:00:19.989 ********** 2026-03-20 15:50:17.276339 | controller | Friday 20 March 2026 15:49:44 +0000 (0:00:00.025) 0:00:19.987 ********** 2026-03-20 15:50:17.276352 | controller | changed: [controller] 2026-03-20 15:50:17.514066 | controller | 2026-03-20 15:50:17.514132 | controller | TASK [ci_setup : Gather version of openshift client _raw_params=oc version --client -o yaml] *** 2026-03-20 15:50:17.514140 | controller | Friday 20 March 2026 15:50:17 +0000 (0:00:32.911) 0:00:52.900 ********** 2026-03-20 15:50:17.514146 | controller | Friday 20 March 2026 15:50:17 +0000 (0:00:32.911) 0:00:52.898 ********** 2026-03-20 15:50:17.514163 | controller | ok: [controller] 2026-03-20 15:50:17.744952 | controller | 2026-03-20 15:50:17.744992 | controller | TASK [ci_setup : Ensure openshift client install path is present path={{ cifmw_ci_setup_oc_install_path }}, state=directory, mode=0755] *** 2026-03-20 15:50:17.745000 | controller | Friday 20 March 2026 15:50:17 +0000 (0:00:00.238) 0:00:53.138 ********** 2026-03-20 15:50:17.745007 | controller | Friday 20 March 2026 15:50:17 +0000 (0:00:00.238) 0:00:53.137 ********** 2026-03-20 15:50:17.745018 | controller | changed: [controller] 2026-03-20 15:50:22.154213 | controller | 2026-03-20 15:50:22.154253 | controller | TASK [ci_setup : Install openshift client src={{ cifmw_ci_setup_openshift_client_download_uri }}/{{ cifmw_ci_setup_openshift_client_version }}/openshift-client-linux.tar.gz, dest={{ cifmw_ci_setup_oc_install_path }}, remote_src=True, mode=0755, creates={{ cifmw_ci_setup_oc_install_path }}/oc] *** 2026-03-20 15:50:22.154264 | controller | Friday 20 March 2026 15:50:17 +0000 (0:00:00.230) 0:00:53.369 ********** 2026-03-20 15:50:22.154269 | controller | Friday 20 March 2026 15:50:17 +0000 (0:00:00.230) 0:00:53.368 ********** 2026-03-20 15:50:22.154279 | controller | changed: [controller] 2026-03-20 15:50:22.179868 | controller | 2026-03-20 15:50:22.179940 | controller | TASK [ci_setup : Add the OC path to cifmw_path if needed cifmw_path={{ cifmw_ci_setup_oc_install_path }}:{{ ansible_env.PATH }}, cacheable=True] *** 2026-03-20 15:50:22.179950 | controller | Friday 20 March 2026 15:50:22 +0000 (0:00:04.409) 0:00:57.778 ********** 2026-03-20 15:50:22.179957 | controller | Friday 20 March 2026 15:50:22 +0000 (0:00:04.409) 0:00:57.777 ********** 2026-03-20 15:50:22.179989 | controller | skipping: [controller] 2026-03-20 15:50:22.489050 | controller | 2026-03-20 15:50:22.489083 | controller | TASK [ci_setup : Create completion file] *************************************** 2026-03-20 15:50:22.489089 | controller | Friday 20 March 2026 15:50:22 +0000 (0:00:00.025) 0:00:57.804 ********** 2026-03-20 15:50:22.489094 | controller | Friday 20 March 2026 15:50:22 +0000 (0:00:00.025) 0:00:57.802 ********** 2026-03-20 15:50:22.489102 | controller | changed: [controller] 2026-03-20 15:50:22.787098 | controller | 2026-03-20 15:50:22.787134 | controller | TASK [ci_setup : Source completion from within .bashrc create=True, mode=0644, path={{ ansible_user_dir }}/.bashrc, block=if [ -f ~/.oc_completion ]; then 2026-03-20 15:50:22.787141 | controller | source ~/.oc_completion 2026-03-20 15:50:22.787147 | controller | fi] *** 2026-03-20 15:50:22.787152 | controller | Friday 20 March 2026 15:50:22 +0000 (0:00:00.309) 0:00:58.113 ********** 2026-03-20 15:50:22.787156 | controller | Friday 20 March 2026 15:50:22 +0000 (0:00:00.309) 0:00:58.112 ********** 2026-03-20 15:50:22.787165 | controller | changed: [controller] 2026-03-20 15:50:22.806052 | controller | 2026-03-20 15:50:22.806086 | controller | TASK [ci_setup : Check rhsm status _raw_params=subscription-manager status] **** 2026-03-20 15:50:22.806093 | controller | Friday 20 March 2026 15:50:22 +0000 (0:00:00.298) 0:00:58.411 ********** 2026-03-20 15:50:22.806098 | controller | Friday 20 March 2026 15:50:22 +0000 (0:00:00.298) 0:00:58.410 ********** 2026-03-20 15:50:22.806106 | controller | skipping: [controller] 2026-03-20 15:50:22.826200 | controller | 2026-03-20 15:50:22.826240 | controller | TASK [ci_setup : Gather the repos to be enabled _repos={{ 2026-03-20 15:50:22.826248 | controller | cifmw_ci_setup_rhel_rhsm_default_repos + 2026-03-20 15:50:22.826255 | controller | (cifmw_ci_setup_rhel_rhsm_extra_repos | default([])) 2026-03-20 15:50:22.826261 | controller | }}] *** 2026-03-20 15:50:22.826267 | controller | Friday 20 March 2026 15:50:22 +0000 (0:00:00.018) 0:00:58.430 ********** 2026-03-20 15:50:22.826274 | controller | Friday 20 March 2026 15:50:22 +0000 (0:00:00.018) 0:00:58.429 ********** 2026-03-20 15:50:22.826286 | controller | skipping: [controller] 2026-03-20 15:50:22.845277 | controller | 2026-03-20 15:50:22.845310 | controller | TASK [ci_setup : Enabling the required repositories. name={{ item }}, state={{ rhsm_repo_state | default('enabled') }}] *** 2026-03-20 15:50:22.845316 | controller | Friday 20 March 2026 15:50:22 +0000 (0:00:00.019) 0:00:58.450 ********** 2026-03-20 15:50:22.845327 | controller | Friday 20 March 2026 15:50:22 +0000 (0:00:00.019) 0:00:58.449 ********** 2026-03-20 15:50:22.845336 | controller | skipping: [controller] 2026-03-20 15:50:22.866288 | controller | 2026-03-20 15:50:22.866329 | controller | TASK [ci_setup : Get current /etc/redhat-release _raw_params=cat /etc/redhat-release] *** 2026-03-20 15:50:22.866337 | controller | Friday 20 March 2026 15:50:22 +0000 (0:00:00.019) 0:00:58.469 ********** 2026-03-20 15:50:22.866345 | controller | Friday 20 March 2026 15:50:22 +0000 (0:00:00.019) 0:00:58.468 ********** 2026-03-20 15:50:22.866358 | controller | skipping: [controller] 2026-03-20 15:50:22.885546 | controller | 2026-03-20 15:50:22.885584 | controller | TASK [ci_setup : Print current /etc/redhat-release msg={{ _current_rh_release.stdout }}] *** 2026-03-20 15:50:22.885591 | controller | Friday 20 March 2026 15:50:22 +0000 (0:00:00.020) 0:00:58.490 ********** 2026-03-20 15:50:22.885596 | controller | Friday 20 March 2026 15:50:22 +0000 (0:00:00.020) 0:00:58.489 ********** 2026-03-20 15:50:22.885604 | controller | skipping: [controller] 2026-03-20 15:50:22.909844 | controller | 2026-03-20 15:50:22.909875 | controller | TASK [ci_setup : Ensure the repos are enabled in the system using yum name={{ item.name }}, baseurl={{ item.baseurl }}, description={{ item.description | default(item.name) }}, gpgcheck={{ item.gpgcheck | default(false) }}, enabled=True, state={{ yum_repo_state | default('present') }}] *** 2026-03-20 15:50:22.909882 | controller | Friday 20 March 2026 15:50:22 +0000 (0:00:00.019) 0:00:58.509 ********** 2026-03-20 15:50:22.909887 | controller | Friday 20 March 2026 15:50:22 +0000 (0:00:00.019) 0:00:58.508 ********** 2026-03-20 15:50:22.909895 | controller | skipping: [controller] 2026-03-20 15:50:23.971197 | controller | 2026-03-20 15:50:23.971265 | controller | TASK [ci_setup : Manage directories path={{ item }}, state={{ directory_state }}, mode=0755, owner={{ ansible_user_id }}, group={{ ansible_user_id }}] *** 2026-03-20 15:50:23.971291 | controller | Friday 20 March 2026 15:50:22 +0000 (0:00:00.024) 0:00:58.534 ********** 2026-03-20 15:50:23.971297 | controller | Friday 20 March 2026 15:50:22 +0000 (0:00:00.024) 0:00:58.532 ********** 2026-03-20 15:50:23.971312 | controller | changed: [controller] => (item=/home/zuul/ci-framework-data/artifacts/manifests/openstack/cr) 2026-03-20 15:50:24.096631 | controller | changed: [controller] => (item=/home/zuul/ci-framework-data/logs) 2026-03-20 15:50:24.096672 | controller | ok: [controller] => (item=/home/zuul/ci-framework-data/tmp) 2026-03-20 15:50:24.096685 | controller | changed: [controller] => (item=/home/zuul/ci-framework-data/volumes) 2026-03-20 15:50:24.096711 | controller | ok: [controller] => (item=/home/zuul/ci-framework-data/artifacts/parameters) 2026-03-20 15:50:24.096720 | controller | 2026-03-20 15:50:24.096726 | controller | TASK [Prepare install_yamls make targets name=install_yamls, apply={'tags': ['bootstrap']}] *** 2026-03-20 15:50:24.096732 | controller | Friday 20 March 2026 15:50:23 +0000 (0:00:01.061) 0:00:59.595 ********** 2026-03-20 15:50:24.096737 | controller | Friday 20 March 2026 15:50:23 +0000 (0:00:01.061) 0:00:59.594 ********** 2026-03-20 15:50:24.096749 | controller | 2026-03-20 15:50:24.769837 | controller | TASK [install_yamls : Ensure directories exist path={{ item }}, state=directory, mode=0755] *** 2026-03-20 15:50:24.769901 | controller | Friday 20 March 2026 15:50:24 +0000 (0:00:00.125) 0:00:59.720 ********** 2026-03-20 15:50:24.769911 | controller | Friday 20 March 2026 15:50:24 +0000 (0:00:00.125) 0:00:59.719 ********** 2026-03-20 15:50:24.769929 | controller | ok: [controller] => (item=/home/zuul/ci-framework-data/artifacts) 2026-03-20 15:50:24.818081 | controller | changed: [controller] => (item=/home/zuul/ci-framework-data/artifacts/roles/install_yamls_makes/tasks) 2026-03-20 15:50:24.818143 | controller | ok: [controller] => (item=/home/zuul/ci-framework-data/artifacts/parameters) 2026-03-20 15:50:24.818152 | controller | 2026-03-20 15:50:24.818160 | controller | TASK [Create variables with local repos based on Zuul items name=install_yamls, tasks_from=zuul_set_operators_repo.yml] *** 2026-03-20 15:50:24.818167 | controller | Friday 20 March 2026 15:50:24 +0000 (0:00:00.672) 0:01:00.393 ********** 2026-03-20 15:50:24.818174 | controller | Friday 20 March 2026 15:50:24 +0000 (0:00:00.672) 0:01:00.392 ********** 2026-03-20 15:50:24.818190 | controller | 2026-03-20 15:50:24.886884 | controller | TASK [install_yamls : Set fact with local repos based on Zuul items cifmw_install_yamls_operators_repo={{ cifmw_install_yamls_operators_repo | default({}) | combine(_repo_operator_info | items2dict) }}] *** 2026-03-20 15:50:24.886941 | controller | Friday 20 March 2026 15:50:24 +0000 (0:00:00.048) 0:01:00.442 ********** 2026-03-20 15:50:24.886948 | controller | Friday 20 March 2026 15:50:24 +0000 (0:00:00.048) 0:01:00.440 ********** 2026-03-20 15:50:24.886963 | controller | ok: [controller] => (item={'branch': 'main', 'change': '354', 'change_url': 'https://github.com/openstack-k8s-operators/watcher-operator/pull/354', 'commit_id': 'ee00c2d330b27d46c48ac29a20680b56ca50df3c', 'patchset': 'ee00c2d330b27d46c48ac29a20680b56ca50df3c', 'project': {'canonical_hostname': 'github.com', 'canonical_name': 'github.com/openstack-k8s-operators/watcher-operator', 'name': 'openstack-k8s-operators/watcher-operator', 'short_name': 'watcher-operator', 'src_dir': 'src/github.com/openstack-k8s-operators/watcher-operator'}, 'topic': None}) 2026-03-20 15:50:24.938350 | controller | 2026-03-20 15:50:24.938457 | controller | TASK [install_yamls : Print helpful data for debugging msg=_repo_operator_name: {{ _repo_operator_name }} 2026-03-20 15:50:24.938509 | controller | _repo_operator_info: {{ _repo_operator_info }} 2026-03-20 15:50:24.938528 | controller | cifmw_install_yamls_operators_repo: {{ cifmw_install_yamls_operators_repo }} 2026-03-20 15:50:24.938538 | controller | ] *** 2026-03-20 15:50:24.938554 | controller | Friday 20 March 2026 15:50:24 +0000 (0:00:00.068) 0:01:00.510 ********** 2026-03-20 15:50:24.938561 | controller | Friday 20 March 2026 15:50:24 +0000 (0:00:00.068) 0:01:00.509 ********** 2026-03-20 15:50:24.938575 | controller | ok: [controller] => (item={'branch': 'main', 'change': '354', 'change_url': 'https://github.com/openstack-k8s-operators/watcher-operator/pull/354', 'commit_id': 'ee00c2d330b27d46c48ac29a20680b56ca50df3c', 'patchset': 'ee00c2d330b27d46c48ac29a20680b56ca50df3c', 'project': {'canonical_hostname': 'github.com', 'canonical_name': 'github.com/openstack-k8s-operators/watcher-operator', 'name': 'openstack-k8s-operators/watcher-operator', 'short_name': 'watcher-operator', 'src_dir': 'src/github.com/openstack-k8s-operators/watcher-operator'}, 'topic': None}) => 2026-03-20 15:50:24.972463 | controller | msg: | 2026-03-20 15:50:24.972501 | controller | _repo_operator_name: watcher 2026-03-20 15:50:24.972509 | controller | _repo_operator_info: [{'key': 'WATCHER_REPO', 'value': '/home/zuul/src/github.com/openstack-k8s-operators/watcher-operator'}, {'key': 'WATCHER_BRANCH', 'value': ''}] 2026-03-20 15:50:24.972516 | controller | cifmw_install_yamls_operators_repo: {'WATCHER_REPO': '/home/zuul/src/github.com/openstack-k8s-operators/watcher-operator', 'WATCHER_BRANCH': ''} 2026-03-20 15:50:24.972523 | controller | 2026-03-20 15:50:24.972528 | controller | TASK [Customize install_yamls devsetup vars if needed name=install_yamls, tasks_from=customize_devsetup_vars.yml] *** 2026-03-20 15:50:24.972534 | controller | Friday 20 March 2026 15:50:24 +0000 (0:00:00.052) 0:01:00.562 ********** 2026-03-20 15:50:24.972540 | controller | Friday 20 March 2026 15:50:24 +0000 (0:00:00.052) 0:01:00.561 ********** 2026-03-20 15:50:24.972551 | controller | 2026-03-20 15:50:24.992530 | controller | TASK [install_yamls : Update opm_version in install_yamls devsetup/vars/default.yaml path={{ cifmw_install_yamls_repo }}/devsetup/vars/default.yaml, regexp=^opm_version:, line=opm_version: {{ cifmw_install_yamls_opm_version }}, state=present] *** 2026-03-20 15:50:24.992570 | controller | Friday 20 March 2026 15:50:24 +0000 (0:00:00.034) 0:01:00.596 ********** 2026-03-20 15:50:24.992578 | controller | Friday 20 March 2026 15:50:24 +0000 (0:00:00.034) 0:01:00.595 ********** 2026-03-20 15:50:24.992590 | controller | skipping: [controller] 2026-03-20 15:50:25.012832 | controller | 2026-03-20 15:50:25.012871 | controller | TASK [install_yamls : Update sdk_version in install_yamls devsetup/vars/default.yaml path={{ cifmw_install_yamls_repo }}/devsetup/vars/default.yaml, regexp=^sdk_version:, line=sdk_version: {{ cifmw_install_yamls_sdk_version }}, state=present] *** 2026-03-20 15:50:25.012881 | controller | Friday 20 March 2026 15:50:24 +0000 (0:00:00.020) 0:01:00.616 ********** 2026-03-20 15:50:25.012888 | controller | Friday 20 March 2026 15:50:24 +0000 (0:00:00.020) 0:01:00.615 ********** 2026-03-20 15:50:25.012899 | controller | skipping: [controller] 2026-03-20 15:50:25.033019 | controller | 2026-03-20 15:50:25.033057 | controller | TASK [install_yamls : Update go_version in install_yamls devsetup/vars/default.yaml path={{ cifmw_install_yamls_repo }}/devsetup/vars/default.yaml, regexp=^go_version:, line=go_version: {{ cifmw_install_yamls_go_version }}, state=present] *** 2026-03-20 15:50:25.033066 | controller | Friday 20 March 2026 15:50:25 +0000 (0:00:00.020) 0:01:00.637 ********** 2026-03-20 15:50:25.033073 | controller | Friday 20 March 2026 15:50:25 +0000 (0:00:00.020) 0:01:00.636 ********** 2026-03-20 15:50:25.033084 | controller | skipping: [controller] 2026-03-20 15:50:25.056686 | controller | 2026-03-20 15:50:25.056744 | controller | TASK [install_yamls : Update kustomize_version in install_yamls devsetup/vars/default.yaml path={{ cifmw_install_yamls_repo }}/devsetup/vars/default.yaml, regexp=^kustomize_version:, line=kustomize_version: {{ cifmw_install_yamls_kustomize_version }}, state=present] *** 2026-03-20 15:50:25.056752 | controller | Friday 20 March 2026 15:50:25 +0000 (0:00:00.020) 0:01:00.657 ********** 2026-03-20 15:50:25.056761 | controller | Friday 20 March 2026 15:50:25 +0000 (0:00:00.020) 0:01:00.656 ********** 2026-03-20 15:50:25.056770 | controller | skipping: [controller] 2026-03-20 15:50:25.164302 | controller | 2026-03-20 15:50:25.164334 | controller | TASK [install_yamls : Compute the cifmw_install_yamls_vars final value _install_yamls_override_vars={{ 2026-03-20 15:50:25.164340 | controller | _install_yamls_override_vars | default({}) | 2026-03-20 15:50:25.164346 | controller | combine(item, recursive=True) 2026-03-20 15:50:25.164351 | controller | }}] *** 2026-03-20 15:50:25.164361 | controller | Friday 20 March 2026 15:50:25 +0000 (0:00:00.023) 0:01:00.681 ********** 2026-03-20 15:50:25.164366 | controller | Friday 20 March 2026 15:50:25 +0000 (0:00:00.023) 0:01:00.679 ********** 2026-03-20 15:50:25.164375 | controller | ok: [controller] => (item={'BMO_SETUP': False}) 2026-03-20 15:50:25.207758 | controller | 2026-03-20 15:50:25.207797 | controller | TASK [install_yamls : Set environment override cifmw_install_yamls_environment fact cifmw_install_yamls_environment={{ 2026-03-20 15:50:25.207804 | controller | _install_yamls_override_vars.keys() | 2026-03-20 15:50:25.207811 | controller | map('upper') | 2026-03-20 15:50:25.207817 | controller | zip(_install_yamls_override_vars.values()) | 2026-03-20 15:50:25.207824 | controller | items2dict(key_name=0, value_name=1) | 2026-03-20 15:50:25.207830 | controller | combine({ 2026-03-20 15:50:25.207836 | controller | 'OUT': cifmw_install_yamls_manifests_dir, 2026-03-20 15:50:25.207842 | controller | 'OUTPUT_DIR': cifmw_install_yamls_edpm_dir, 2026-03-20 15:50:25.207848 | controller | 'CHECKOUT_FROM_OPENSTACK_REF': cifmw_install_yamls_checkout_openstack_ref, 2026-03-20 15:50:25.207855 | controller | 'OPENSTACK_K8S_BRANCH': (zuul is defined and not zuul.branch |regex_search('master|antelope|rhos')) | ternary(zuul.branch, 'main') 2026-03-20 15:50:25.207861 | controller | }) | 2026-03-20 15:50:25.207867 | controller | combine(install_yamls_operators_repos) 2026-03-20 15:50:25.207878 | controller | }}, cacheable=True] *** 2026-03-20 15:50:25.207884 | controller | Friday 20 March 2026 15:50:25 +0000 (0:00:00.107) 0:01:00.788 ********** 2026-03-20 15:50:25.207890 | controller | Friday 20 March 2026 15:50:25 +0000 (0:00:00.107) 0:01:00.787 ********** 2026-03-20 15:50:25.207902 | controller | ok: [controller] 2026-03-20 15:50:25.837998 | controller | 2026-03-20 15:50:25.838075 | controller | TASK [install_yamls : Get environment structure base_path={{ cifmw_install_yamls_repo }}] *** 2026-03-20 15:50:25.838083 | controller | Friday 20 March 2026 15:50:25 +0000 (0:00:00.043) 0:01:00.831 ********** 2026-03-20 15:50:25.838090 | controller | Friday 20 March 2026 15:50:25 +0000 (0:00:00.043) 0:01:00.830 ********** 2026-03-20 15:50:25.838110 | controller | ok: [controller] 2026-03-20 15:50:26.113219 | controller | 2026-03-20 15:50:26.113288 | controller | TASK [install_yamls : Ensure Output directory exists path={{ cifmw_install_yamls_out_dir }}, state=directory, mode=0755] *** 2026-03-20 15:50:26.113295 | controller | Friday 20 March 2026 15:50:25 +0000 (0:00:00.629) 0:01:01.461 ********** 2026-03-20 15:50:26.113300 | controller | Friday 20 March 2026 15:50:25 +0000 (0:00:00.629) 0:01:01.460 ********** 2026-03-20 15:50:26.113315 | controller | ok: [controller] 2026-03-20 15:50:26.160103 | controller | 2026-03-20 15:50:26.160209 | controller | TASK [install_yamls : Ensure user cifmw_install_yamls_vars contains existing Makefile variables that=_cifmw_install_yamls_unmatched_vars | length == 0, msg=cifmw_install_yamls_vars contains a variable that is not defined in install_yamls Makefile nor cifmw_install_yamls_whitelisted_vars: {{ _cifmw_install_yamls_unmatched_vars | join(', ')}}, quiet=True] *** 2026-03-20 15:50:26.160224 | controller | Friday 20 March 2026 15:50:26 +0000 (0:00:00.275) 0:01:01.737 ********** 2026-03-20 15:50:26.160232 | controller | Friday 20 March 2026 15:50:26 +0000 (0:00:00.275) 0:01:01.736 ********** 2026-03-20 15:50:26.160250 | controller | ok: [controller] 2026-03-20 15:50:26.842510 | controller | 2026-03-20 15:50:26.842612 | controller | TASK [install_yamls : Generate /home/zuul/ci-framework-data/artifacts/install_yamls.sh dest={{ cifmw_install_yamls_out_dir }}/{{ cifmw_install_yamls_envfile }}, content={% for k,v in cifmw_install_yamls_environment.items() %} 2026-03-20 15:50:26.842622 | controller | export {{ k }}={{ v }} 2026-03-20 15:50:26.842630 | controller | {% endfor %}, mode=0644] *** 2026-03-20 15:50:26.842636 | controller | Friday 20 March 2026 15:50:26 +0000 (0:00:00.046) 0:01:01.784 ********** 2026-03-20 15:50:26.842643 | controller | Friday 20 March 2026 15:50:26 +0000 (0:00:00.046) 0:01:01.783 ********** 2026-03-20 15:50:26.842664 | controller | changed: [controller] 2026-03-20 15:50:26.871674 | controller | 2026-03-20 15:50:26.871778 | controller | TASK [install_yamls : Set install_yamls default values cifmw_install_yamls_defaults={{ get_makefiles_env_output.makefiles_values | combine(cifmw_install_yamls_environment) }}, cacheable=True] *** 2026-03-20 15:50:26.871799 | controller | Friday 20 March 2026 15:50:26 +0000 (0:00:00.682) 0:01:02.466 ********** 2026-03-20 15:50:26.871806 | controller | Friday 20 March 2026 15:50:26 +0000 (0:00:00.682) 0:01:02.465 ********** 2026-03-20 15:50:26.871824 | controller | ok: [controller] 2026-03-20 15:50:26.892839 | controller | 2026-03-20 15:50:26.892905 | controller | TASK [install_yamls : Show the env structure var=cifmw_install_yamls_environment] *** 2026-03-20 15:50:26.892911 | controller | Friday 20 March 2026 15:50:26 +0000 (0:00:00.029) 0:01:02.495 ********** 2026-03-20 15:50:26.892916 | controller | Friday 20 March 2026 15:50:26 +0000 (0:00:00.029) 0:01:02.494 ********** 2026-03-20 15:50:26.892931 | controller | ok: [controller] => 2026-03-20 15:50:26.916529 | controller | cifmw_install_yamls_environment: 2026-03-20 15:50:26.916590 | controller | BMO_SETUP: false 2026-03-20 15:50:26.916598 | controller | CHECKOUT_FROM_OPENSTACK_REF: 'true' 2026-03-20 15:50:26.916605 | controller | OPENSTACK_K8S_BRANCH: main 2026-03-20 15:50:26.916611 | controller | OUT: /home/zuul/ci-framework-data/artifacts/manifests 2026-03-20 15:50:26.916621 | controller | OUTPUT_DIR: /home/zuul/ci-framework-data/artifacts/edpm 2026-03-20 15:50:26.916627 | controller | WATCHER_BRANCH: '' 2026-03-20 15:50:26.916633 | controller | WATCHER_REPO: /home/zuul/src/github.com/openstack-k8s-operators/watcher-operator 2026-03-20 15:50:26.916640 | controller | 2026-03-20 15:50:26.916646 | controller | TASK [install_yamls : Show the env structure defaults var=cifmw_install_yamls_defaults] *** 2026-03-20 15:50:26.916652 | controller | Friday 20 March 2026 15:50:26 +0000 (0:00:00.021) 0:01:02.517 ********** 2026-03-20 15:50:26.916659 | controller | Friday 20 March 2026 15:50:26 +0000 (0:00:00.021) 0:01:02.515 ********** 2026-03-20 15:50:26.916672 | controller | ok: [controller] => 2026-03-20 15:50:26.916678 | controller | cifmw_install_yamls_defaults: 2026-03-20 15:50:26.916684 | controller | ADOPTED_EXTERNAL_NETWORK: 172.21.1.0/24 2026-03-20 15:50:26.916706 | controller | ADOPTED_INTERNALAPI_NETWORK: 172.17.1.0/24 2026-03-20 15:50:26.916728 | controller | ADOPTED_STORAGEMGMT_NETWORK: 172.20.1.0/24 2026-03-20 15:50:26.916735 | controller | ADOPTED_STORAGE_NETWORK: 172.18.1.0/24 2026-03-20 15:50:26.916741 | controller | ADOPTED_TENANT_NETWORK: 172.9.1.0/24 2026-03-20 15:50:26.916747 | controller | BARBICAN: config/samples/barbican_v1beta1_barbican.yaml 2026-03-20 15:50:26.916753 | controller | BARBICAN_BRANCH: main 2026-03-20 15:50:26.916759 | controller | BARBICAN_COMMIT_HASH: '' 2026-03-20 15:50:26.916766 | controller | BARBICAN_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/barbican-operator/config/samples/barbican_v1beta1_barbican.yaml 2026-03-20 15:50:26.916772 | controller | BARBICAN_DEPL_IMG: unused 2026-03-20 15:50:26.916778 | controller | BARBICAN_IMG: quay.io/openstack-k8s-operators/barbican-operator-index:latest 2026-03-20 15:50:26.916785 | controller | BARBICAN_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/barbican-operator/kuttl-test.yaml 2026-03-20 15:50:26.916792 | controller | BARBICAN_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/barbican-operator/test/kuttl/tests 2026-03-20 15:50:26.916797 | controller | BARBICAN_KUTTL_NAMESPACE: barbican-kuttl-tests 2026-03-20 15:50:26.916803 | controller | BARBICAN_REPO: https://github.com/openstack-k8s-operators/barbican-operator.git 2026-03-20 15:50:26.916809 | controller | BARBICAN_SERVICE_ENABLED: 'true' 2026-03-20 15:50:26.916815 | controller | BARBICAN_SIMPLE_CRYPTO_ENCRYPTION_KEY: sEFmdFjDUqRM2VemYslV5yGNWjokioJXsg8Nrlc3drU= 2026-03-20 15:50:26.916821 | controller | BAREMETAL_BRANCH: main 2026-03-20 15:50:26.916828 | controller | BAREMETAL_COMMIT_HASH: '' 2026-03-20 15:50:26.916834 | controller | BAREMETAL_IMG: quay.io/openstack-k8s-operators/openstack-baremetal-operator-index:latest 2026-03-20 15:50:26.916841 | controller | BAREMETAL_OS_CONTAINER_IMG: '' 2026-03-20 15:50:26.916847 | controller | BAREMETAL_OS_IMG: '' 2026-03-20 15:50:26.916853 | controller | BAREMETAL_OS_IMG_TYPE: '' 2026-03-20 15:50:26.916860 | controller | BAREMETAL_REPO: https://github.com/openstack-k8s-operators/openstack-baremetal-operator.git 2026-03-20 15:50:26.916866 | controller | BAREMETAL_TIMEOUT: 20m 2026-03-20 15:50:26.916872 | controller | BASH_IMG: quay.io/openstack-k8s-operators/bash:latest 2026-03-20 15:50:26.916878 | controller | BGP_ASN: '64999' 2026-03-20 15:50:26.916890 | controller | BGP_LEAF_1: 100.65.4.1 2026-03-20 15:50:26.916896 | controller | BGP_LEAF_2: 100.64.4.1 2026-03-20 15:50:26.916903 | controller | BGP_OVN_ROUTING: 'false' 2026-03-20 15:50:26.916909 | controller | BGP_PEER_ASN: '64999' 2026-03-20 15:50:26.916915 | controller | BGP_SOURCE_IP: 172.30.4.2 2026-03-20 15:50:26.916921 | controller | BGP_SOURCE_IP6: f00d:f00d:f00d:f00d:f00d:f00d:f00d:42 2026-03-20 15:50:26.916927 | controller | BMAAS_BRIDGE_IPV4_PREFIX: 172.20.1.2/24 2026-03-20 15:50:26.916934 | controller | BMAAS_BRIDGE_IPV6_PREFIX: fd00:bbbb::2/64 2026-03-20 15:50:26.916940 | controller | BMAAS_INSTANCE_DISK_SIZE: '20' 2026-03-20 15:50:26.916946 | controller | BMAAS_INSTANCE_MEMORY: '4096' 2026-03-20 15:50:26.916952 | controller | BMAAS_INSTANCE_NAME_PREFIX: crc-bmaas 2026-03-20 15:50:26.916957 | controller | BMAAS_INSTANCE_NET_MODEL: virtio 2026-03-20 15:50:26.916964 | controller | BMAAS_INSTANCE_OS_VARIANT: centos-stream9 2026-03-20 15:50:26.916971 | controller | BMAAS_INSTANCE_VCPUS: '2' 2026-03-20 15:50:26.916978 | controller | BMAAS_INSTANCE_VIRT_TYPE: kvm 2026-03-20 15:50:26.916984 | controller | BMAAS_IPV4: 'true' 2026-03-20 15:50:26.916990 | controller | BMAAS_IPV6: 'false' 2026-03-20 15:50:26.916997 | controller | BMAAS_LIBVIRT_USER: sushyemu 2026-03-20 15:50:26.917003 | controller | BMAAS_METALLB_ADDRESS_POOL: 172.20.1.64/26 2026-03-20 15:50:26.917010 | controller | BMAAS_METALLB_POOL_NAME: baremetal 2026-03-20 15:50:26.917017 | controller | BMAAS_NETWORK_IPV4_PREFIX: 172.20.1.1/24 2026-03-20 15:50:26.917023 | controller | BMAAS_NETWORK_IPV6_PREFIX: fd00:bbbb::1/64 2026-03-20 15:50:26.917029 | controller | BMAAS_NETWORK_NAME: crc-bmaas 2026-03-20 15:50:26.917036 | controller | BMAAS_NODE_COUNT: '1' 2026-03-20 15:50:26.917042 | controller | BMAAS_OCP_INSTANCE_NAME: crc 2026-03-20 15:50:26.917049 | controller | BMAAS_REDFISH_PASSWORD: password 2026-03-20 15:50:26.917055 | controller | BMAAS_REDFISH_USERNAME: admin 2026-03-20 15:50:26.917062 | controller | BMAAS_ROUTE_LIBVIRT_NETWORKS: crc-bmaas,crc,default 2026-03-20 15:50:26.917068 | controller | BMAAS_SUSHY_EMULATOR_DRIVER: libvirt 2026-03-20 15:50:26.917075 | controller | BMAAS_SUSHY_EMULATOR_IMAGE: quay.io/metal3-io/sushy-tools:latest 2026-03-20 15:50:26.917082 | controller | BMAAS_SUSHY_EMULATOR_NAMESPACE: sushy-emulator 2026-03-20 15:50:26.917088 | controller | BMAAS_SUSHY_EMULATOR_OS_CLIENT_CONFIG_FILE: /etc/openstack/clouds.yaml 2026-03-20 15:50:26.917095 | controller | BMAAS_SUSHY_EMULATOR_OS_CLOUD: openstack 2026-03-20 15:50:26.917101 | controller | BMH_NAMESPACE: openstack 2026-03-20 15:50:26.917108 | controller | BMO_BRANCH: release-0.9 2026-03-20 15:50:26.917114 | controller | BMO_CLEANUP: 'true' 2026-03-20 15:50:26.917120 | controller | BMO_COMMIT_HASH: '' 2026-03-20 15:50:26.917127 | controller | BMO_IPA_BASEURI: https://tarballs.opendev.org/openstack/ironic-python-agent/dib 2026-03-20 15:50:26.917133 | controller | BMO_IPA_BRANCH: stable/2024.1 2026-03-20 15:50:26.917140 | controller | BMO_IPA_FLAVOR: centos9 2026-03-20 15:50:26.917146 | controller | BMO_IPA_INSECURE: 'false' 2026-03-20 15:50:26.917153 | controller | BMO_IRONIC_HOST: 192.168.122.10 2026-03-20 15:50:26.917159 | controller | BMO_PROVISIONING_INTERFACE: '' 2026-03-20 15:50:26.917166 | controller | BMO_REPO: https://github.com/metal3-io/baremetal-operator 2026-03-20 15:50:26.917172 | controller | BMO_SETUP: false 2026-03-20 15:50:26.917179 | controller | BMO_SETUP_ROUTE_REPLACE: 'true' 2026-03-20 15:50:26.917185 | controller | BM_CTLPLANE_INTERFACE: enp1s0 2026-03-20 15:50:26.917192 | controller | BM_INSTANCE_MEMORY: '8192' 2026-03-20 15:50:26.917198 | controller | BM_INSTANCE_NAME_PREFIX: edpm-compute-baremetal 2026-03-20 15:50:26.917204 | controller | BM_INSTANCE_NAME_SUFFIX: '0' 2026-03-20 15:50:26.917211 | controller | BM_NETWORK_NAME: default 2026-03-20 15:50:26.917217 | controller | BM_NODE_COUNT: '1' 2026-03-20 15:50:26.917224 | controller | BM_ROOT_PASSWORD: '' 2026-03-20 15:50:26.917230 | controller | BM_ROOT_PASSWORD_SECRET: '' 2026-03-20 15:50:26.917237 | controller | CEILOMETER_CENTRAL_DEPL_IMG: unused 2026-03-20 15:50:26.917243 | controller | CEILOMETER_NOTIFICATION_DEPL_IMG: unused 2026-03-20 15:50:26.917249 | controller | CEPH_BRANCH: release-1.15 2026-03-20 15:50:26.917256 | controller | CEPH_CLIENT: /home/zuul/ci-framework-data/artifacts/manifests/operator/rook/deploy/examples/toolbox.yaml 2026-03-20 15:50:26.917267 | controller | CEPH_COMMON: /home/zuul/ci-framework-data/artifacts/manifests/operator/rook/deploy/examples/common.yaml 2026-03-20 15:50:26.917278 | controller | CEPH_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/rook/deploy/examples/cluster-test.yaml 2026-03-20 15:50:26.917284 | controller | CEPH_CRDS: /home/zuul/ci-framework-data/artifacts/manifests/operator/rook/deploy/examples/crds.yaml 2026-03-20 15:50:26.917291 | controller | CEPH_IMG: quay.io/ceph/demo:latest-squid 2026-03-20 15:50:26.917297 | controller | CEPH_OP: /home/zuul/ci-framework-data/artifacts/manifests/operator/rook/deploy/examples/operator-openshift.yaml 2026-03-20 15:50:26.917304 | controller | CEPH_REPO: https://github.com/rook/rook.git 2026-03-20 15:50:26.917310 | controller | CERTMANAGER_TIMEOUT: 300s 2026-03-20 15:50:26.917316 | controller | CHECKOUT_FROM_OPENSTACK_REF: 'true' 2026-03-20 15:50:26.917323 | controller | CINDER: config/samples/cinder_v1beta1_cinder.yaml 2026-03-20 15:50:26.917329 | controller | CINDERAPI_DEPL_IMG: unused 2026-03-20 15:50:26.917336 | controller | CINDERBKP_DEPL_IMG: unused 2026-03-20 15:50:26.917342 | controller | CINDERSCH_DEPL_IMG: unused 2026-03-20 15:50:26.917349 | controller | CINDERVOL_DEPL_IMG: unused 2026-03-20 15:50:26.917355 | controller | CINDER_BRANCH: main 2026-03-20 15:50:26.917362 | controller | CINDER_COMMIT_HASH: '' 2026-03-20 15:50:26.917368 | controller | CINDER_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/cinder-operator/config/samples/cinder_v1beta1_cinder.yaml 2026-03-20 15:50:26.917375 | controller | CINDER_IMG: quay.io/openstack-k8s-operators/cinder-operator-index:latest 2026-03-20 15:50:26.917382 | controller | CINDER_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/cinder-operator/kuttl-test.yaml 2026-03-20 15:50:26.917388 | controller | CINDER_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/cinder-operator/test/kuttl/tests 2026-03-20 15:50:26.917395 | controller | CINDER_KUTTL_NAMESPACE: cinder-kuttl-tests 2026-03-20 15:50:26.917401 | controller | CINDER_REPO: https://github.com/openstack-k8s-operators/cinder-operator.git 2026-03-20 15:50:26.917408 | controller | CLEANUP_DIR_CMD: rm -Rf 2026-03-20 15:50:26.917414 | controller | CRC_BGP_NIC_1_MAC: '52:54:00:11:11:11' 2026-03-20 15:50:26.917421 | controller | CRC_BGP_NIC_2_MAC: '52:54:00:11:11:12' 2026-03-20 15:50:26.917432 | controller | CRC_HTTPS_PROXY: '' 2026-03-20 15:50:26.917438 | controller | CRC_HTTP_PROXY: '' 2026-03-20 15:50:26.917445 | controller | CRC_STORAGE_NAMESPACE: crc-storage 2026-03-20 15:50:26.917452 | controller | CRC_STORAGE_RETRIES: '3' 2026-03-20 15:50:26.917459 | controller | CRC_URL: '''https://developers.redhat.com/content-gateway/rest/mirror/pub/openshift-v4/clients/crc/latest/crc-linux-amd64.tar.xz''' 2026-03-20 15:50:26.917465 | controller | CRC_VERSION: latest 2026-03-20 15:50:26.917472 | controller | DATAPLANE_ANSIBLE_SECRET: dataplane-ansible-ssh-private-key-secret 2026-03-20 15:50:26.917478 | controller | DATAPLANE_ANSIBLE_USER: '' 2026-03-20 15:50:26.917485 | controller | DATAPLANE_COMPUTE_IP: 192.168.122.100 2026-03-20 15:50:26.917491 | controller | DATAPLANE_CONTAINER_PREFIX: openstack 2026-03-20 15:50:26.917498 | controller | DATAPLANE_CONTAINER_TAG: current-podified 2026-03-20 15:50:26.917539 | controller | DATAPLANE_CUSTOM_SERVICE_RUNNER_IMG: quay.io/openstack-k8s-operators/openstack-ansibleee-runner:latest 2026-03-20 15:50:26.917545 | controller | DATAPLANE_DEFAULT_GW: 192.168.122.1 2026-03-20 15:50:26.917552 | controller | DATAPLANE_EXTRA_NOVA_CONFIG_FILE: /dev/null 2026-03-20 15:50:26.917559 | controller | DATAPLANE_GROWVOLS_ARGS: /=8GB /tmp=1GB /home=1GB /var=100% 2026-03-20 15:50:26.917566 | controller | DATAPLANE_KUSTOMIZE_SCENARIO: preprovisioned 2026-03-20 15:50:26.917572 | controller | DATAPLANE_NETWORKER_IP: 192.168.122.200 2026-03-20 15:50:26.917578 | controller | DATAPLANE_NETWORK_INTERFACE_NAME: eth0 2026-03-20 15:50:26.917584 | controller | DATAPLANE_NOVA_NFS_PATH: '' 2026-03-20 15:50:26.917591 | controller | DATAPLANE_NTP_SERVER: pool.ntp.org 2026-03-20 15:50:26.917597 | controller | DATAPLANE_PLAYBOOK: osp.edpm.download_cache 2026-03-20 15:50:26.917603 | controller | DATAPLANE_REGISTRY_URL: quay.io/podified-antelope-centos9 2026-03-20 15:50:26.917609 | controller | DATAPLANE_RUNNER_IMG: '' 2026-03-20 15:50:26.917615 | controller | DATAPLANE_SERVER_ROLE: compute 2026-03-20 15:50:26.917626 | controller | DATAPLANE_SSHD_ALLOWED_RANGES: '[''192.168.122.0/24'']' 2026-03-20 15:50:26.917632 | controller | DATAPLANE_TIMEOUT: 30m 2026-03-20 15:50:26.917638 | controller | DATAPLANE_TLS_ENABLED: 'true' 2026-03-20 15:50:26.917644 | controller | DATAPLANE_TOTAL_NETWORKER_NODES: '1' 2026-03-20 15:50:26.917650 | controller | DATAPLANE_TOTAL_NODES: '1' 2026-03-20 15:50:26.917656 | controller | DBSERVICE: galera 2026-03-20 15:50:26.917663 | controller | DESIGNATE: config/samples/designate_v1beta1_designate.yaml 2026-03-20 15:50:26.917669 | controller | DESIGNATE_BRANCH: main 2026-03-20 15:50:26.917675 | controller | DESIGNATE_COMMIT_HASH: '' 2026-03-20 15:50:26.917681 | controller | DESIGNATE_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/designate-operator/config/samples/designate_v1beta1_designate.yaml 2026-03-20 15:50:26.917688 | controller | DESIGNATE_IMG: quay.io/openstack-k8s-operators/designate-operator-index:latest 2026-03-20 15:50:26.917720 | controller | DESIGNATE_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/designate-operator/kuttl-test.yaml 2026-03-20 15:50:26.917727 | controller | DESIGNATE_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/designate-operator/test/kuttl/tests 2026-03-20 15:50:26.917734 | controller | DESIGNATE_KUTTL_NAMESPACE: designate-kuttl-tests 2026-03-20 15:50:26.917740 | controller | DESIGNATE_REPO: https://github.com/openstack-k8s-operators/designate-operator.git 2026-03-20 15:50:26.917747 | controller | DNSDATA: config/samples/network_v1beta1_dnsdata.yaml 2026-03-20 15:50:26.917757 | controller | DNSDATA_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/infra-operator/config/samples/network_v1beta1_dnsdata.yaml 2026-03-20 15:50:26.917764 | controller | DNSMASQ: config/samples/network_v1beta1_dnsmasq.yaml 2026-03-20 15:50:26.917770 | controller | DNSMASQ_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/infra-operator/config/samples/network_v1beta1_dnsmasq.yaml 2026-03-20 15:50:26.917777 | controller | DNS_DEPL_IMG: unused 2026-03-20 15:50:26.917783 | controller | DNS_DOMAIN: localdomain 2026-03-20 15:50:26.917794 | controller | DOWNLOAD_TOOLS_SELECTION: all 2026-03-20 15:50:26.917801 | controller | EDPM_ATTACH_EXTNET: 'true' 2026-03-20 15:50:26.917808 | controller | EDPM_COMPUTE_ADDITIONAL_HOST_ROUTES: '''[]''' 2026-03-20 15:50:26.917814 | controller | EDPM_COMPUTE_ADDITIONAL_NETWORKS: '''[]''' 2026-03-20 15:50:26.917821 | controller | EDPM_COMPUTE_CELLS: '1' 2026-03-20 15:50:26.917827 | controller | EDPM_COMPUTE_CEPH_ENABLED: 'true' 2026-03-20 15:50:26.917834 | controller | EDPM_COMPUTE_CEPH_NOVA: 'true' 2026-03-20 15:50:26.917840 | controller | EDPM_COMPUTE_DHCP_AGENT_ENABLED: 'true' 2026-03-20 15:50:26.917847 | controller | EDPM_COMPUTE_SRIOV_ENABLED: 'true' 2026-03-20 15:50:26.917853 | controller | EDPM_COMPUTE_SUFFIX: '0' 2026-03-20 15:50:26.917860 | controller | EDPM_CONFIGURE_DEFAULT_ROUTE: 'true' 2026-03-20 15:50:26.917866 | controller | EDPM_CONFIGURE_HUGEPAGES: 'false' 2026-03-20 15:50:26.917873 | controller | EDPM_CONFIGURE_NETWORKING: 'true' 2026-03-20 15:50:26.917879 | controller | EDPM_FIRSTBOOT_EXTRA: /tmp/edpm-firstboot-extra 2026-03-20 15:50:26.917886 | controller | EDPM_NETWORKER_SUFFIX: '0' 2026-03-20 15:50:26.917892 | controller | EDPM_TOTAL_NETWORKERS: '1' 2026-03-20 15:50:26.917899 | controller | EDPM_TOTAL_NODES: '1' 2026-03-20 15:50:26.917905 | controller | GALERA_REPLICAS: '' 2026-03-20 15:50:26.917912 | controller | GENERATE_SSH_KEYS: 'true' 2026-03-20 15:50:26.917919 | controller | GIT_CLONE_OPTS: '' 2026-03-20 15:50:26.917925 | controller | GLANCE: config/samples/glance_v1beta1_glance.yaml 2026-03-20 15:50:26.917935 | controller | GLANCEAPI_DEPL_IMG: unused 2026-03-20 15:50:26.917941 | controller | GLANCE_BRANCH: main 2026-03-20 15:50:26.917948 | controller | GLANCE_COMMIT_HASH: '' 2026-03-20 15:50:26.917954 | controller | GLANCE_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/glance-operator/config/samples/glance_v1beta1_glance.yaml 2026-03-20 15:50:26.917961 | controller | GLANCE_IMG: quay.io/openstack-k8s-operators/glance-operator-index:latest 2026-03-20 15:50:26.917967 | controller | GLANCE_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/glance-operator/kuttl-test.yaml 2026-03-20 15:50:26.917974 | controller | GLANCE_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/glance-operator/test/kuttl/tests 2026-03-20 15:50:26.917984 | controller | GLANCE_KUTTL_NAMESPACE: glance-kuttl-tests 2026-03-20 15:50:26.917991 | controller | GLANCE_REPO: https://github.com/openstack-k8s-operators/glance-operator.git 2026-03-20 15:50:26.917997 | controller | HEAT: config/samples/heat_v1beta1_heat.yaml 2026-03-20 15:50:26.918004 | controller | HEATAPI_DEPL_IMG: unused 2026-03-20 15:50:26.918010 | controller | HEATCFNAPI_DEPL_IMG: unused 2026-03-20 15:50:26.918017 | controller | HEATENGINE_DEPL_IMG: unused 2026-03-20 15:50:26.918023 | controller | HEAT_AUTH_ENCRYPTION_KEY: 767c3ed056cbaa3b9dfedb8c6f825bf0 2026-03-20 15:50:26.918030 | controller | HEAT_BRANCH: main 2026-03-20 15:50:26.918036 | controller | HEAT_COMMIT_HASH: '' 2026-03-20 15:50:26.918046 | controller | HEAT_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/heat-operator/config/samples/heat_v1beta1_heat.yaml 2026-03-20 15:50:26.918053 | controller | HEAT_IMG: quay.io/openstack-k8s-operators/heat-operator-index:latest 2026-03-20 15:50:26.918059 | controller | HEAT_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/heat-operator/kuttl-test.yaml 2026-03-20 15:50:26.918067 | controller | HEAT_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/heat-operator/test/kuttl/tests 2026-03-20 15:50:26.918074 | controller | HEAT_KUTTL_NAMESPACE: heat-kuttl-tests 2026-03-20 15:50:26.918080 | controller | HEAT_REPO: https://github.com/openstack-k8s-operators/heat-operator.git 2026-03-20 15:50:26.918087 | controller | HEAT_SERVICE_ENABLED: 'true' 2026-03-20 15:50:26.918094 | controller | HORIZON: config/samples/horizon_v1beta1_horizon.yaml 2026-03-20 15:50:26.918100 | controller | HORIZON_BRANCH: main 2026-03-20 15:50:26.918106 | controller | HORIZON_COMMIT_HASH: '' 2026-03-20 15:50:26.918113 | controller | HORIZON_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/horizon-operator/config/samples/horizon_v1beta1_horizon.yaml 2026-03-20 15:50:26.918119 | controller | HORIZON_DEPL_IMG: unused 2026-03-20 15:50:26.918126 | controller | HORIZON_IMG: quay.io/openstack-k8s-operators/horizon-operator-index:latest 2026-03-20 15:50:26.918132 | controller | HORIZON_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/horizon-operator/kuttl-test.yaml 2026-03-20 15:50:26.918140 | controller | HORIZON_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/horizon-operator/test/kuttl/tests 2026-03-20 15:50:26.918146 | controller | HORIZON_KUTTL_NAMESPACE: horizon-kuttl-tests 2026-03-20 15:50:26.918153 | controller | HORIZON_REPO: https://github.com/openstack-k8s-operators/horizon-operator.git 2026-03-20 15:50:26.918160 | controller | INFRA_BRANCH: main 2026-03-20 15:50:26.918166 | controller | INFRA_COMMIT_HASH: '' 2026-03-20 15:50:26.918173 | controller | INFRA_IMG: quay.io/openstack-k8s-operators/infra-operator-index:latest 2026-03-20 15:50:26.918179 | controller | INFRA_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/infra-operator/kuttl-test.yaml 2026-03-20 15:50:26.918186 | controller | INFRA_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/infra-operator/test/kuttl/tests 2026-03-20 15:50:26.918192 | controller | INFRA_KUTTL_NAMESPACE: infra-kuttl-tests 2026-03-20 15:50:26.918199 | controller | INFRA_REPO: https://github.com/openstack-k8s-operators/infra-operator.git 2026-03-20 15:50:26.918205 | controller | INSTALL_CERT_MANAGER: 'true' 2026-03-20 15:50:26.918212 | controller | INSTALL_NMSTATE: true || false 2026-03-20 15:50:26.918218 | controller | INSTALL_NNCP: true || false 2026-03-20 15:50:26.918225 | controller | INTERNALAPI_HOST_ROUTES: '' 2026-03-20 15:50:26.918231 | controller | IPV6_LAB_IPV4_NETWORK_IPADDRESS: 172.30.0.1/24 2026-03-20 15:50:26.918238 | controller | IPV6_LAB_IPV6_NETWORK_IPADDRESS: fd00:abcd:abcd:fc00::1/64 2026-03-20 15:50:26.918244 | controller | IPV6_LAB_LIBVIRT_STORAGE_POOL: default 2026-03-20 15:50:26.918250 | controller | IPV6_LAB_MANAGE_FIREWALLD: 'true' 2026-03-20 15:50:26.918256 | controller | IPV6_LAB_NAT64_HOST_IPV4: 172.30.0.2/24 2026-03-20 15:50:26.918262 | controller | IPV6_LAB_NAT64_HOST_IPV6: fd00:abcd:abcd:fc00::2/64 2026-03-20 15:50:26.918268 | controller | IPV6_LAB_NAT64_INSTANCE_NAME: nat64-router 2026-03-20 15:50:26.918273 | controller | IPV6_LAB_NAT64_IPV6_NETWORK: fd00:abcd:abcd:fc00::/64 2026-03-20 15:50:26.918289 | controller | IPV6_LAB_NAT64_TAYGA_DYNAMIC_POOL: 192.168.255.0/24 2026-03-20 15:50:26.918296 | controller | IPV6_LAB_NAT64_TAYGA_IPV4: 192.168.255.1 2026-03-20 15:50:26.918304 | controller | IPV6_LAB_NAT64_TAYGA_IPV6: fd00:abcd:abcd:fc00::3 2026-03-20 15:50:26.918310 | controller | IPV6_LAB_NAT64_TAYGA_IPV6_PREFIX: fd00:abcd:abcd:fcff::/96 2026-03-20 15:50:26.918317 | controller | IPV6_LAB_NAT64_UPDATE_PACKAGES: 'false' 2026-03-20 15:50:26.918323 | controller | IPV6_LAB_NETWORK_NAME: nat64 2026-03-20 15:50:26.918329 | controller | IPV6_LAB_SNO_CLUSTER_NETWORK: fd00:abcd:0::/48 2026-03-20 15:50:26.918336 | controller | IPV6_LAB_SNO_HOST_IP: fd00:abcd:abcd:fc00::11 2026-03-20 15:50:26.918342 | controller | IPV6_LAB_SNO_HOST_PREFIX: '64' 2026-03-20 15:50:26.918348 | controller | IPV6_LAB_SNO_INSTANCE_NAME: sno 2026-03-20 15:50:26.918355 | controller | IPV6_LAB_SNO_MACHINE_NETWORK: fd00:abcd:abcd:fc00::/64 2026-03-20 15:50:26.918361 | controller | IPV6_LAB_SNO_OCP_MIRROR_URL: https://mirror.openshift.com/pub/openshift-v4/clients/ocp 2026-03-20 15:50:26.918367 | controller | IPV6_LAB_SNO_OCP_VERSION: latest-4.14 2026-03-20 15:50:26.918373 | controller | IPV6_LAB_SNO_SERVICE_NETWORK: fd00:abcd:abcd:fc03::/112 2026-03-20 15:50:26.918379 | controller | IPV6_LAB_SSH_PUB_KEY: /home/zuul/.ssh/id_rsa.pub 2026-03-20 15:50:26.918385 | controller | IPV6_LAB_WORK_DIR: /home/zuul/.ipv6lab 2026-03-20 15:50:26.918391 | controller | IRONIC: config/samples/ironic_v1beta1_ironic.yaml 2026-03-20 15:50:26.918396 | controller | IRONICAPI_DEPL_IMG: unused 2026-03-20 15:50:26.918403 | controller | IRONICCON_DEPL_IMG: unused 2026-03-20 15:50:26.918409 | controller | IRONICINS_DEPL_IMG: unused 2026-03-20 15:50:26.918415 | controller | IRONICNAG_DEPL_IMG: unused 2026-03-20 15:50:26.918424 | controller | IRONICPXE_DEPL_IMG: unused 2026-03-20 15:50:26.918430 | controller | IRONIC_BRANCH: main 2026-03-20 15:50:26.918436 | controller | IRONIC_COMMIT_HASH: '' 2026-03-20 15:50:26.918442 | controller | IRONIC_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/ironic-operator/config/samples/ironic_v1beta1_ironic.yaml 2026-03-20 15:50:26.918448 | controller | IRONIC_IMAGE: quay.io/metal3-io/ironic 2026-03-20 15:50:26.918454 | controller | IRONIC_IMAGE_TAG: release-24.1 2026-03-20 15:50:26.918460 | controller | IRONIC_IMG: quay.io/openstack-k8s-operators/ironic-operator-index:latest 2026-03-20 15:50:26.918466 | controller | IRONIC_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/ironic-operator/kuttl-test.yaml 2026-03-20 15:50:26.918472 | controller | IRONIC_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/ironic-operator/test/kuttl/tests 2026-03-20 15:50:26.918478 | controller | IRONIC_KUTTL_NAMESPACE: ironic-kuttl-tests 2026-03-20 15:50:26.918484 | controller | IRONIC_REPO: https://github.com/openstack-k8s-operators/ironic-operator.git 2026-03-20 15:50:26.918490 | controller | KEYSTONEAPI: config/samples/keystone_v1beta1_keystoneapi.yaml 2026-03-20 15:50:26.918496 | controller | KEYSTONEAPI_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/keystone-operator/config/samples/keystone_v1beta1_keystoneapi.yaml 2026-03-20 15:50:26.918504 | controller | KEYSTONEAPI_DEPL_IMG: unused 2026-03-20 15:50:26.918510 | controller | KEYSTONE_BRANCH: main 2026-03-20 15:50:26.918516 | controller | KEYSTONE_COMMIT_HASH: '' 2026-03-20 15:50:26.918522 | controller | KEYSTONE_FEDERATION_CLIENT_SECRET: COX8bmlKAWn56XCGMrKQJj7dgHNAOl6f 2026-03-20 15:50:26.918528 | controller | KEYSTONE_FEDERATION_CRYPTO_PASSPHRASE: openstack 2026-03-20 15:50:26.918534 | controller | KEYSTONE_IMG: quay.io/openstack-k8s-operators/keystone-operator-index:latest 2026-03-20 15:50:26.918540 | controller | KEYSTONE_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/keystone-operator/kuttl-test.yaml 2026-03-20 15:50:26.918546 | controller | KEYSTONE_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/keystone-operator/test/kuttl/tests 2026-03-20 15:50:26.918552 | controller | KEYSTONE_KUTTL_NAMESPACE: keystone-kuttl-tests 2026-03-20 15:50:26.918558 | controller | KEYSTONE_REPO: https://github.com/openstack-k8s-operators/keystone-operator.git 2026-03-20 15:50:26.918565 | controller | KUBEADMIN_PWD: '12345678' 2026-03-20 15:50:26.918571 | controller | LIBVIRT_SECRET: libvirt-secret 2026-03-20 15:50:26.918580 | controller | LOKI_DEPLOY_MODE: openshift-network 2026-03-20 15:50:26.918586 | controller | LOKI_DEPLOY_NAMESPACE: netobserv 2026-03-20 15:50:26.918593 | controller | LOKI_DEPLOY_SIZE: 1x.demo 2026-03-20 15:50:26.918599 | controller | LOKI_NAMESPACE: openshift-operators-redhat 2026-03-20 15:50:26.918605 | controller | LOKI_OPERATOR_GROUP: openshift-operators-redhat-loki 2026-03-20 15:50:26.918611 | controller | LOKI_SUBSCRIPTION: loki-operator 2026-03-20 15:50:26.918618 | controller | LVMS_CR: '1' 2026-03-20 15:50:26.918624 | controller | MANILA: config/samples/manila_v1beta1_manila.yaml 2026-03-20 15:50:26.918630 | controller | MANILAAPI_DEPL_IMG: unused 2026-03-20 15:50:26.918636 | controller | MANILASCH_DEPL_IMG: unused 2026-03-20 15:50:26.918642 | controller | MANILASHARE_DEPL_IMG: unused 2026-03-20 15:50:26.918648 | controller | MANILA_BRANCH: main 2026-03-20 15:50:26.918654 | controller | MANILA_COMMIT_HASH: '' 2026-03-20 15:50:26.918660 | controller | MANILA_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/manila-operator/config/samples/manila_v1beta1_manila.yaml 2026-03-20 15:50:26.918667 | controller | MANILA_IMG: quay.io/openstack-k8s-operators/manila-operator-index:latest 2026-03-20 15:50:26.918673 | controller | MANILA_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/manila-operator/kuttl-test.yaml 2026-03-20 15:50:26.918679 | controller | MANILA_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/manila-operator/test/kuttl/tests 2026-03-20 15:50:26.918685 | controller | MANILA_KUTTL_NAMESPACE: manila-kuttl-tests 2026-03-20 15:50:26.918711 | controller | MANILA_REPO: https://github.com/openstack-k8s-operators/manila-operator.git 2026-03-20 15:50:26.918724 | controller | MANILA_SERVICE_ENABLED: 'true' 2026-03-20 15:50:26.918731 | controller | MARIADB: config/samples/mariadb_v1beta1_galera.yaml 2026-03-20 15:50:26.918738 | controller | MARIADB_BRANCH: main 2026-03-20 15:50:26.918745 | controller | MARIADB_CHAINSAW_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/mariadb-operator/test/chainsaw/config.yaml 2026-03-20 15:50:26.918757 | controller | MARIADB_CHAINSAW_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/mariadb-operator/test/chainsaw/tests 2026-03-20 15:50:26.918764 | controller | MARIADB_CHAINSAW_NAMESPACE: mariadb-chainsaw-tests 2026-03-20 15:50:26.918771 | controller | MARIADB_COMMIT_HASH: '' 2026-03-20 15:50:26.918777 | controller | MARIADB_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/mariadb-operator/config/samples/mariadb_v1beta1_galera.yaml 2026-03-20 15:50:26.918783 | controller | MARIADB_DEPL_IMG: unused 2026-03-20 15:50:26.918789 | controller | MARIADB_IMG: quay.io/openstack-k8s-operators/mariadb-operator-index:latest 2026-03-20 15:50:26.918795 | controller | MARIADB_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/mariadb-operator/kuttl-test.yaml 2026-03-20 15:50:26.918802 | controller | MARIADB_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/mariadb-operator/test/kuttl/tests 2026-03-20 15:50:26.918808 | controller | MARIADB_KUTTL_NAMESPACE: mariadb-kuttl-tests 2026-03-20 15:50:26.918814 | controller | MARIADB_REPO: https://github.com/openstack-k8s-operators/mariadb-operator.git 2026-03-20 15:50:26.918820 | controller | MEMCACHED: config/samples/memcached_v1beta1_memcached.yaml 2026-03-20 15:50:26.918826 | controller | MEMCACHED_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/infra-operator/config/samples/memcached_v1beta1_memcached.yaml 2026-03-20 15:50:26.918832 | controller | MEMCACHED_DEPL_IMG: unused 2026-03-20 15:50:26.918843 | controller | METADATA_SHARED_SECRET: '1234567842' 2026-03-20 15:50:26.918850 | controller | METALLB_IPV6_POOL: fd00:aaaa::80-fd00:aaaa::90 2026-03-20 15:50:26.918856 | controller | METALLB_POOL: 192.168.122.80-192.168.122.90 2026-03-20 15:50:26.918865 | controller | MICROSHIFT: '0' 2026-03-20 15:50:26.918871 | controller | MIRROR_INSECURE: 'true' 2026-03-20 15:50:26.918877 | controller | MIRROR_NAMESPACE: openstack-mirror 2026-03-20 15:50:26.918882 | controller | MIRROR_REGISTRY_HOST: default-route-openshift-image-registry.apps-crc.testing 2026-03-20 15:50:26.918888 | controller | NAMESPACE: openstack 2026-03-20 15:50:26.918894 | controller | NETCONFIG: config/samples/network_v1beta1_netconfig.yaml 2026-03-20 15:50:26.918904 | controller | NETCONFIG_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/infra-operator/config/samples/network_v1beta1_netconfig.yaml 2026-03-20 15:50:26.918909 | controller | NETCONFIG_DEPL_IMG: unused 2026-03-20 15:50:26.918915 | controller | NETOBSERV_DEPLOY_NAMESPACE: netobserv 2026-03-20 15:50:26.918921 | controller | NETOBSERV_NAMESPACE: openshift-netobserv-operator 2026-03-20 15:50:26.918927 | controller | NETOBSERV_OPERATOR_GROUP: openshift-netobserv-operator-net 2026-03-20 15:50:26.918933 | controller | NETOBSERV_SUBSCRIPTION: netobserv-operator 2026-03-20 15:50:26.918939 | controller | NETWORK_BGP: 'false' 2026-03-20 15:50:26.918945 | controller | NETWORK_DESIGNATE_ADDRESS_PREFIX: 172.28.0 2026-03-20 15:50:26.918951 | controller | NETWORK_DESIGNATE_EXT_ADDRESS_PREFIX: 172.50.0 2026-03-20 15:50:26.918957 | controller | NETWORK_INTERNALAPI_ADDRESS_PREFIX: 172.17.0 2026-03-20 15:50:26.918963 | controller | NETWORK_ISOLATION: 'true' 2026-03-20 15:50:26.918969 | controller | NETWORK_ISOLATION_INSTANCE_NAME: crc 2026-03-20 15:50:26.918975 | controller | NETWORK_ISOLATION_IPV4: 'true' 2026-03-20 15:50:26.918981 | controller | NETWORK_ISOLATION_IPV4_ADDRESS: 172.16.1.1/24 2026-03-20 15:50:26.918987 | controller | NETWORK_ISOLATION_IPV4_NAT: 'true' 2026-03-20 15:50:26.918993 | controller | NETWORK_ISOLATION_IPV6: 'false' 2026-03-20 15:50:26.919000 | controller | NETWORK_ISOLATION_IPV6_ADDRESS: fd00:aaaa::1/64 2026-03-20 15:50:26.919006 | controller | NETWORK_ISOLATION_IP_ADDRESS: 192.168.122.10 2026-03-20 15:50:26.919012 | controller | NETWORK_ISOLATION_MAC: '52:54:00:11:11:10' 2026-03-20 15:50:26.919018 | controller | NETWORK_ISOLATION_NETWORK_NAME: net-iso 2026-03-20 15:50:26.919025 | controller | NETWORK_ISOLATION_NET_NAME: default 2026-03-20 15:50:26.919031 | controller | NETWORK_ISOLATION_USE_DEFAULT_NETWORK: 'true' 2026-03-20 15:50:26.919038 | controller | NETWORK_MTU: '1500' 2026-03-20 15:50:26.919044 | controller | NETWORK_STORAGEMGMT_ADDRESS_PREFIX: 172.20.0 2026-03-20 15:50:26.919051 | controller | NETWORK_STORAGE_ADDRESS_PREFIX: 172.18.0 2026-03-20 15:50:26.919057 | controller | NETWORK_STORAGE_MACVLAN: '' 2026-03-20 15:50:26.919064 | controller | NETWORK_TENANT_ADDRESS_PREFIX: 172.19.0 2026-03-20 15:50:26.919070 | controller | NETWORK_VLAN_START: '20' 2026-03-20 15:50:26.919077 | controller | NETWORK_VLAN_STEP: '1' 2026-03-20 15:50:26.919083 | controller | NEUTRONAPI: config/samples/neutron_v1beta1_neutronapi.yaml 2026-03-20 15:50:26.919089 | controller | NEUTRONAPI_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/neutron-operator/config/samples/neutron_v1beta1_neutronapi.yaml 2026-03-20 15:50:26.919096 | controller | NEUTRONAPI_DEPL_IMG: unused 2026-03-20 15:50:26.919102 | controller | NEUTRON_BRANCH: main 2026-03-20 15:50:26.919108 | controller | NEUTRON_COMMIT_HASH: '' 2026-03-20 15:50:26.919115 | controller | NEUTRON_IMG: quay.io/openstack-k8s-operators/neutron-operator-index:latest 2026-03-20 15:50:26.919121 | controller | NEUTRON_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/neutron-operator/kuttl-test.yaml 2026-03-20 15:50:26.919126 | controller | NEUTRON_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/neutron-operator/test/kuttl/tests 2026-03-20 15:50:26.919132 | controller | NEUTRON_KUTTL_NAMESPACE: neutron-kuttl-tests 2026-03-20 15:50:26.919139 | controller | NEUTRON_REPO: https://github.com/openstack-k8s-operators/neutron-operator.git 2026-03-20 15:50:26.919147 | controller | NFS_HOME: /home/nfs 2026-03-20 15:50:26.919153 | controller | NMSTATE_NAMESPACE: openshift-nmstate 2026-03-20 15:50:26.919159 | controller | NMSTATE_OPERATOR_GROUP: openshift-nmstate-tn6k8 2026-03-20 15:50:26.919165 | controller | NMSTATE_SUBSCRIPTION: kubernetes-nmstate-operator 2026-03-20 15:50:26.919171 | controller | NNCP_ADDITIONAL_HOST_ROUTES: '' 2026-03-20 15:50:26.919177 | controller | NNCP_BGP_1_INTERFACE: enp7s0 2026-03-20 15:50:26.919184 | controller | NNCP_BGP_1_IP_ADDRESS: 100.65.4.2 2026-03-20 15:50:26.919190 | controller | NNCP_BGP_2_INTERFACE: enp8s0 2026-03-20 15:50:26.919197 | controller | NNCP_BGP_2_IP_ADDRESS: 100.64.4.2 2026-03-20 15:50:26.919202 | controller | NNCP_BRIDGE: ospbr 2026-03-20 15:50:26.919208 | controller | NNCP_CLEANUP_TIMEOUT: 120s 2026-03-20 15:50:26.919214 | controller | NNCP_CTLPLANE_IPV6_ADDRESS_PREFIX: 'fd00:aaaa::' 2026-03-20 15:50:26.919227 | controller | NNCP_CTLPLANE_IPV6_ADDRESS_SUFFIX: '10' 2026-03-20 15:50:26.919237 | controller | NNCP_CTLPLANE_IP_ADDRESS_PREFIX: 192.168.122 2026-03-20 15:50:26.919244 | controller | NNCP_CTLPLANE_IP_ADDRESS_SUFFIX: '10' 2026-03-20 15:50:26.919250 | controller | NNCP_DNS_SERVER: 192.168.122.1 2026-03-20 15:50:26.919257 | controller | NNCP_DNS_SERVER_IPV6: fd00:aaaa::1 2026-03-20 15:50:26.919263 | controller | NNCP_GATEWAY: 192.168.122.1 2026-03-20 15:50:26.919269 | controller | NNCP_GATEWAY_IPV6: fd00:aaaa::1 2026-03-20 15:50:26.919275 | controller | NNCP_INTERFACE: enp6s0 2026-03-20 15:50:26.919282 | controller | NNCP_NODES: '' 2026-03-20 15:50:26.919288 | controller | NNCP_TIMEOUT: 240s 2026-03-20 15:50:26.919295 | controller | NOVA: config/samples/nova_v1beta1_nova_collapsed_cell.yaml 2026-03-20 15:50:26.919301 | controller | NOVA_BRANCH: main 2026-03-20 15:50:26.919307 | controller | NOVA_COMMIT_HASH: '' 2026-03-20 15:50:26.919312 | controller | NOVA_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/nova-operator/config/samples/nova_v1beta1_nova_collapsed_cell.yaml 2026-03-20 15:50:26.919319 | controller | NOVA_IMG: quay.io/openstack-k8s-operators/nova-operator-index:latest 2026-03-20 15:50:26.919325 | controller | NOVA_REPO: https://github.com/openstack-k8s-operators/nova-operator.git 2026-03-20 15:50:26.919332 | controller | NUMBER_OF_INSTANCES: '1' 2026-03-20 15:50:26.919338 | controller | OCP_NETWORK_NAME: crc 2026-03-20 15:50:26.919345 | controller | OCTAVIA: config/samples/octavia_v1beta1_octavia.yaml 2026-03-20 15:50:26.919351 | controller | OCTAVIA_BRANCH: main 2026-03-20 15:50:26.919359 | controller | OCTAVIA_COMMIT_HASH: '' 2026-03-20 15:50:26.919365 | controller | OCTAVIA_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/octavia-operator/config/samples/octavia_v1beta1_octavia.yaml 2026-03-20 15:50:26.919371 | controller | OCTAVIA_IMG: quay.io/openstack-k8s-operators/octavia-operator-index:latest 2026-03-20 15:50:26.919378 | controller | OCTAVIA_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/octavia-operator/kuttl-test.yaml 2026-03-20 15:50:26.919384 | controller | OCTAVIA_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/octavia-operator/test/kuttl/tests 2026-03-20 15:50:26.919391 | controller | OCTAVIA_KUTTL_NAMESPACE: octavia-kuttl-tests 2026-03-20 15:50:26.919398 | controller | OCTAVIA_REPO: https://github.com/openstack-k8s-operators/octavia-operator.git 2026-03-20 15:50:26.919404 | controller | OKD: 'false' 2026-03-20 15:50:26.919411 | controller | OPENSTACK_BRANCH: main 2026-03-20 15:50:26.919417 | controller | OPENSTACK_BUNDLE_IMG: quay.io/openstack-k8s-operators/openstack-operator-bundle:latest 2026-03-20 15:50:26.919423 | controller | OPENSTACK_COMMIT_HASH: '' 2026-03-20 15:50:26.919430 | controller | OPENSTACK_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/openstack-operator/config/samples/core_v1beta1_openstackcontrolplane_galera_network_isolation.yaml 2026-03-20 15:50:26.919437 | controller | OPENSTACK_CRDS_DIR: openstack_crds 2026-03-20 15:50:26.919443 | controller | OPENSTACK_CTLPLANE: config/samples/core_v1beta1_openstackcontrolplane_galera_network_isolation.yaml 2026-03-20 15:50:26.919450 | controller | OPENSTACK_IMG: quay.io/openstack-k8s-operators/openstack-operator-index:latest 2026-03-20 15:50:26.919456 | controller | OPENSTACK_K8S_BRANCH: main 2026-03-20 15:50:26.919463 | controller | OPENSTACK_K8S_TAG: latest 2026-03-20 15:50:26.919469 | controller | OPENSTACK_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/openstack-operator/kuttl-test.yaml 2026-03-20 15:50:26.919476 | controller | OPENSTACK_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/openstack-operator/test/kuttl/tests 2026-03-20 15:50:26.919482 | controller | OPENSTACK_KUTTL_NAMESPACE: openstack-kuttl-tests 2026-03-20 15:50:26.919489 | controller | OPENSTACK_NEUTRON_CUSTOM_CONF: '' 2026-03-20 15:50:26.919495 | controller | OPENSTACK_REPO: https://github.com/openstack-k8s-operators/openstack-operator.git 2026-03-20 15:50:26.919503 | controller | OPENSTACK_STORAGE_BUNDLE_IMG: quay.io/openstack-k8s-operators/openstack-operator-storage-bundle:latest 2026-03-20 15:50:26.919510 | controller | OPERATOR_BASE_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator 2026-03-20 15:50:26.919519 | controller | OPERATOR_CHANNEL: '' 2026-03-20 15:50:26.919525 | controller | OPERATOR_NAMESPACE: openstack-operators 2026-03-20 15:50:26.919531 | controller | OPERATOR_SOURCE: '' 2026-03-20 15:50:26.919537 | controller | OPERATOR_SOURCE_NAMESPACE: '' 2026-03-20 15:50:26.919547 | controller | OUT: /home/zuul/ci-framework-data/artifacts/manifests 2026-03-20 15:50:26.919553 | controller | OUTPUT_DIR: /home/zuul/ci-framework-data/artifacts/edpm 2026-03-20 15:50:26.919559 | controller | OVNCONTROLLER: config/samples/ovn_v1beta1_ovncontroller.yaml 2026-03-20 15:50:26.919565 | controller | OVNCONTROLLER_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/ovn-operator/config/samples/ovn_v1beta1_ovncontroller.yaml 2026-03-20 15:50:26.919572 | controller | OVNCONTROLLER_NMAP: 'true' 2026-03-20 15:50:26.919578 | controller | OVNDBS: config/samples/ovn_v1beta1_ovndbcluster.yaml 2026-03-20 15:50:26.919587 | controller | OVNDBS_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/ovn-operator/config/samples/ovn_v1beta1_ovndbcluster.yaml 2026-03-20 15:50:26.919593 | controller | OVNNORTHD: config/samples/ovn_v1beta1_ovnnorthd.yaml 2026-03-20 15:50:26.919600 | controller | OVNNORTHD_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/ovn-operator/config/samples/ovn_v1beta1_ovnnorthd.yaml 2026-03-20 15:50:26.919606 | controller | OVN_BRANCH: main 2026-03-20 15:50:26.919612 | controller | OVN_COMMIT_HASH: '' 2026-03-20 15:50:26.919618 | controller | OVN_IMG: quay.io/openstack-k8s-operators/ovn-operator-index:latest 2026-03-20 15:50:26.919624 | controller | OVN_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/ovn-operator/kuttl-test.yaml 2026-03-20 15:50:26.919632 | controller | OVN_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/ovn-operator/test/kuttl/tests 2026-03-20 15:50:26.919638 | controller | OVN_KUTTL_NAMESPACE: ovn-kuttl-tests 2026-03-20 15:50:26.919645 | controller | OVN_REPO: https://github.com/openstack-k8s-operators/ovn-operator.git 2026-03-20 15:50:26.919651 | controller | PASSWORD: '12345678' 2026-03-20 15:50:26.919662 | controller | PLACEMENTAPI: config/samples/placement_v1beta1_placementapi.yaml 2026-03-20 15:50:26.926103 | controller | PLACEMENTAPI_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/placement-operator/config/samples/placement_v1beta1_placementapi.yaml 2026-03-20 15:50:26.926132 | controller | PLACEMENTAPI_DEPL_IMG: unused 2026-03-20 15:50:26.926139 | controller | PLACEMENT_BRANCH: main 2026-03-20 15:50:26.926146 | controller | PLACEMENT_COMMIT_HASH: '' 2026-03-20 15:50:26.926152 | controller | PLACEMENT_IMG: quay.io/openstack-k8s-operators/placement-operator-index:latest 2026-03-20 15:50:26.926158 | controller | PLACEMENT_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/placement-operator/kuttl-test.yaml 2026-03-20 15:50:26.926164 | controller | PLACEMENT_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/placement-operator/test/kuttl/tests 2026-03-20 15:50:26.926169 | controller | PLACEMENT_KUTTL_NAMESPACE: placement-kuttl-tests 2026-03-20 15:50:26.926175 | controller | PLACEMENT_REPO: https://github.com/openstack-k8s-operators/placement-operator.git 2026-03-20 15:50:26.926181 | controller | PULL_SECRET: /home/zuul/pull-secret.txt 2026-03-20 15:50:26.926186 | controller | RABBITMQ: docs/examples/default-security-context/rabbitmq.yaml 2026-03-20 15:50:26.926192 | controller | RABBITMQ_BRANCH: patches 2026-03-20 15:50:26.926197 | controller | RABBITMQ_COMMIT_HASH: '' 2026-03-20 15:50:26.926203 | controller | RABBITMQ_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/rabbitmq-operator/docs/examples/default-security-context/rabbitmq.yaml 2026-03-20 15:50:26.926209 | controller | RABBITMQ_DEPL_IMG: unused 2026-03-20 15:50:26.926214 | controller | RABBITMQ_IMG: quay.io/openstack-k8s-operators/rabbitmq-cluster-operator-index:latest 2026-03-20 15:50:26.926220 | controller | RABBITMQ_REPO: https://github.com/openstack-k8s-operators/rabbitmq-cluster-operator.git 2026-03-20 15:50:26.926225 | controller | REDHAT_OPERATORS: 'false' 2026-03-20 15:50:26.926231 | controller | REDIS: config/samples/redis_v1beta1_redis.yaml 2026-03-20 15:50:26.926237 | controller | REDIS_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/infra-operator-redis/config/samples/redis_v1beta1_redis.yaml 2026-03-20 15:50:26.926247 | controller | REDIS_DEPL_IMG: unused 2026-03-20 15:50:26.926253 | controller | RH_REGISTRY_PWD: '' 2026-03-20 15:50:26.926258 | controller | RH_REGISTRY_USER: '' 2026-03-20 15:50:26.926264 | controller | SECRET: osp-secret 2026-03-20 15:50:26.926269 | controller | SG_CORE_DEPL_IMG: unused 2026-03-20 15:50:26.926275 | controller | STANDALONE_COMPUTE_DRIVER: libvirt 2026-03-20 15:50:26.926280 | controller | STANDALONE_EXTERNAL_NET_PREFFIX: 172.21.0 2026-03-20 15:50:26.926286 | controller | STANDALONE_INTERNALAPI_NET_PREFIX: 172.17.0 2026-03-20 15:50:26.926291 | controller | STANDALONE_STORAGEMGMT_NET_PREFIX: 172.20.0 2026-03-20 15:50:26.926297 | controller | STANDALONE_STORAGE_NET_PREFIX: 172.18.0 2026-03-20 15:50:26.926302 | controller | STANDALONE_TENANT_NET_PREFIX: 172.19.0 2026-03-20 15:50:26.926308 | controller | STORAGEMGMT_HOST_ROUTES: '' 2026-03-20 15:50:26.926313 | controller | STORAGE_CLASS: local-storage 2026-03-20 15:50:26.926319 | controller | STORAGE_HOST_ROUTES: '' 2026-03-20 15:50:26.926324 | controller | SWIFT: config/samples/swift_v1beta1_swift.yaml 2026-03-20 15:50:26.926330 | controller | SWIFT_BRANCH: main 2026-03-20 15:50:26.926335 | controller | SWIFT_COMMIT_HASH: '' 2026-03-20 15:50:26.926340 | controller | SWIFT_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/swift-operator/config/samples/swift_v1beta1_swift.yaml 2026-03-20 15:50:26.926346 | controller | SWIFT_IMG: quay.io/openstack-k8s-operators/swift-operator-index:latest 2026-03-20 15:50:26.926351 | controller | SWIFT_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/swift-operator/kuttl-test.yaml 2026-03-20 15:50:26.926357 | controller | SWIFT_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/swift-operator/test/kuttl/tests 2026-03-20 15:50:26.926362 | controller | SWIFT_KUTTL_NAMESPACE: swift-kuttl-tests 2026-03-20 15:50:26.926367 | controller | SWIFT_REPO: https://github.com/openstack-k8s-operators/swift-operator.git 2026-03-20 15:50:26.926373 | controller | TELEMETRY: config/samples/telemetry_v1beta1_telemetry.yaml 2026-03-20 15:50:26.926380 | controller | TELEMETRY_BRANCH: main 2026-03-20 15:50:26.926386 | controller | TELEMETRY_COMMIT_HASH: '' 2026-03-20 15:50:26.926392 | controller | TELEMETRY_CR: /home/zuul/ci-framework-data/artifacts/manifests/operator/telemetry-operator/config/samples/telemetry_v1beta1_telemetry.yaml 2026-03-20 15:50:26.926398 | controller | TELEMETRY_IMG: quay.io/openstack-k8s-operators/telemetry-operator-index:latest 2026-03-20 15:50:26.926404 | controller | TELEMETRY_KUTTL_BASEDIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/telemetry-operator 2026-03-20 15:50:26.926410 | controller | TELEMETRY_KUTTL_CONF: /home/zuul/ci-framework-data/artifacts/manifests/operator/telemetry-operator/kuttl-test.yaml 2026-03-20 15:50:26.926415 | controller | TELEMETRY_KUTTL_DIR: /home/zuul/ci-framework-data/artifacts/manifests/operator/telemetry-operator/test/kuttl/suites 2026-03-20 15:50:26.926420 | controller | TELEMETRY_KUTTL_NAMESPACE: telemetry-kuttl-tests 2026-03-20 15:50:26.926425 | controller | TELEMETRY_KUTTL_RELPATH: test/kuttl/suites 2026-03-20 15:50:26.926431 | controller | TELEMETRY_REPO: https://github.com/openstack-k8s-operators/telemetry-operator.git 2026-03-20 15:50:26.926436 | controller | TENANT_HOST_ROUTES: '' 2026-03-20 15:50:26.926442 | controller | TIMEOUT: 300s 2026-03-20 15:50:26.926452 | controller | TLS_ENABLED: 'false' 2026-03-20 15:50:26.926458 | controller | WATCHER_BRANCH: '' 2026-03-20 15:50:26.926464 | controller | WATCHER_REPO: /home/zuul/src/github.com/openstack-k8s-operators/watcher-operator 2026-03-20 15:50:26.926469 | controller | tripleo_deploy: 'export REGISTRY_PWD:' 2026-03-20 15:50:26.926479 | controller | 2026-03-20 15:50:27.267853 | controller | TASK [install_yamls : Generate make targets install_yamls_path={{ cifmw_install_yamls_repo }}, output_directory={{ cifmw_install_yamls_tasks_out }}] *** 2026-03-20 15:50:27.267922 | controller | Friday 20 March 2026 15:50:26 +0000 (0:00:00.033) 0:01:02.550 ********** 2026-03-20 15:50:27.267929 | controller | Friday 20 March 2026 15:50:26 +0000 (0:00:00.033) 0:01:02.549 ********** 2026-03-20 15:50:27.267939 | controller | ok: [controller] 2026-03-20 15:50:27.287773 | controller | 2026-03-20 15:50:27.287828 | controller | TASK [install_yamls : Debug generate_make module var=cifmw_generate_makes] ***** 2026-03-20 15:50:27.287845 | controller | Friday 20 March 2026 15:50:27 +0000 (0:00:00.341) 0:01:02.891 ********** 2026-03-20 15:50:27.287850 | controller | Friday 20 March 2026 15:50:27 +0000 (0:00:00.341) 0:01:02.890 ********** 2026-03-20 15:50:27.287867 | controller | ok: [controller] => 2026-03-20 15:50:27.287874 | controller | cifmw_generate_makes: 2026-03-20 15:50:27.287880 | controller | changed: false 2026-03-20 15:50:27.287886 | controller | debug: 2026-03-20 15:50:27.287892 | controller | /home/zuul/src/github.com/openstack-k8s-operators/install_yamls/Makefile: 2026-03-20 15:50:27.287897 | controller | - all 2026-03-20 15:50:27.287903 | controller | - help 2026-03-20 15:50:27.287908 | controller | - cleanup 2026-03-20 15:50:27.287914 | controller | - deploy_cleanup 2026-03-20 15:50:27.287920 | controller | - wait 2026-03-20 15:50:27.287925 | controller | - crc_storage 2026-03-20 15:50:27.287931 | controller | - crc_storage_cleanup 2026-03-20 15:50:27.287937 | controller | - crc_storage_release 2026-03-20 15:50:27.287942 | controller | - crc_storage_with_retries 2026-03-20 15:50:27.287949 | controller | - crc_storage_cleanup_with_retries 2026-03-20 15:50:27.287954 | controller | - operator_namespace 2026-03-20 15:50:27.287960 | controller | - namespace 2026-03-20 15:50:27.287965 | controller | - namespace_cleanup 2026-03-20 15:50:27.287970 | controller | - input 2026-03-20 15:50:27.287974 | controller | - input_cleanup 2026-03-20 15:50:27.287978 | controller | - crc_bmo_setup 2026-03-20 15:50:27.287982 | controller | - crc_bmo_cleanup 2026-03-20 15:50:27.287986 | controller | - mirror_registry 2026-03-20 15:50:27.287991 | controller | - mirror_registry_secure 2026-03-20 15:50:27.287995 | controller | - mirror_registry_ca 2026-03-20 15:50:27.287999 | controller | - mirror_registry_cleanup 2026-03-20 15:50:27.288003 | controller | - openstack_prep 2026-03-20 15:50:27.288007 | controller | - openstack 2026-03-20 15:50:27.288012 | controller | - openstack_wait 2026-03-20 15:50:27.288016 | controller | - openstack_init 2026-03-20 15:50:27.288020 | controller | - openstack_cleanup 2026-03-20 15:50:27.288024 | controller | - openstack_repo 2026-03-20 15:50:27.288028 | controller | - openstack_deploy_prep 2026-03-20 15:50:27.288033 | controller | - openstack_deploy 2026-03-20 15:50:27.288037 | controller | - openstack_wait_deploy 2026-03-20 15:50:27.288041 | controller | - openstack_deploy_cleanup 2026-03-20 15:50:27.288045 | controller | - openstack_update_run 2026-03-20 15:50:27.288049 | controller | - update_services 2026-03-20 15:50:27.288054 | controller | - update_system 2026-03-20 15:50:27.288058 | controller | - openstack_patch_version 2026-03-20 15:50:27.288063 | controller | - edpm_deploy_generate_keys 2026-03-20 15:50:27.288067 | controller | - edpm_patch_ansible_runner_image 2026-03-20 15:50:27.288071 | controller | - edpm_deploy_prep 2026-03-20 15:50:27.288075 | controller | - edpm_deploy_cleanup 2026-03-20 15:50:27.288079 | controller | - edpm_deploy 2026-03-20 15:50:27.288084 | controller | - edpm_deploy_baremetal_prep 2026-03-20 15:50:27.288088 | controller | - edpm_deploy_baremetal 2026-03-20 15:50:27.288092 | controller | - edpm_wait_deploy_baremetal 2026-03-20 15:50:27.288096 | controller | - edpm_wait_deploy 2026-03-20 15:50:27.288100 | controller | - edpm_register_dns 2026-03-20 15:50:27.288104 | controller | - edpm_nova_discover_hosts 2026-03-20 15:50:27.288108 | controller | - openstack_crds 2026-03-20 15:50:27.288112 | controller | - openstack_crds_cleanup 2026-03-20 15:50:27.288117 | controller | - edpm_deploy_networker_prep 2026-03-20 15:50:27.288121 | controller | - edpm_deploy_networker_cleanup 2026-03-20 15:50:27.288125 | controller | - edpm_deploy_networker 2026-03-20 15:50:27.288129 | controller | - infra_prep 2026-03-20 15:50:27.288133 | controller | - infra 2026-03-20 15:50:27.288137 | controller | - infra_cleanup 2026-03-20 15:50:27.288141 | controller | - dns_deploy_prep 2026-03-20 15:50:27.288146 | controller | - dns_deploy 2026-03-20 15:50:27.288150 | controller | - dns_deploy_cleanup 2026-03-20 15:50:27.288154 | controller | - netconfig_deploy_prep 2026-03-20 15:50:27.288158 | controller | - netconfig_deploy 2026-03-20 15:50:27.288162 | controller | - netconfig_deploy_cleanup 2026-03-20 15:50:27.288166 | controller | - memcached_deploy_prep 2026-03-20 15:50:27.288170 | controller | - memcached_deploy 2026-03-20 15:50:27.288177 | controller | - memcached_deploy_cleanup 2026-03-20 15:50:27.288181 | controller | - keystone_prep 2026-03-20 15:50:27.288186 | controller | - keystone 2026-03-20 15:50:27.288190 | controller | - keystone_cleanup 2026-03-20 15:50:27.288194 | controller | - keystone_deploy_prep 2026-03-20 15:50:27.288198 | controller | - keystone_deploy 2026-03-20 15:50:27.288202 | controller | - keystone_deploy_cleanup 2026-03-20 15:50:27.288206 | controller | - barbican_prep 2026-03-20 15:50:27.288210 | controller | - barbican 2026-03-20 15:50:27.288214 | controller | - barbican_cleanup 2026-03-20 15:50:27.288218 | controller | - barbican_deploy_prep 2026-03-20 15:50:27.288223 | controller | - barbican_deploy 2026-03-20 15:50:27.288227 | controller | - barbican_deploy_validate 2026-03-20 15:50:27.288231 | controller | - barbican_deploy_cleanup 2026-03-20 15:50:27.288235 | controller | - mariadb 2026-03-20 15:50:27.288239 | controller | - mariadb_cleanup 2026-03-20 15:50:27.288243 | controller | - mariadb_deploy_prep 2026-03-20 15:50:27.288247 | controller | - mariadb_deploy 2026-03-20 15:50:27.288252 | controller | - mariadb_deploy_cleanup 2026-03-20 15:50:27.288256 | controller | - placement_prep 2026-03-20 15:50:27.288260 | controller | - placement 2026-03-20 15:50:27.288264 | controller | - placement_cleanup 2026-03-20 15:50:27.288269 | controller | - placement_deploy_prep 2026-03-20 15:50:27.288275 | controller | - placement_deploy 2026-03-20 15:50:27.288286 | controller | - placement_deploy_cleanup 2026-03-20 15:50:27.288292 | controller | - glance_prep 2026-03-20 15:50:27.288298 | controller | - glance 2026-03-20 15:50:27.288303 | controller | - glance_cleanup 2026-03-20 15:50:27.288308 | controller | - glance_deploy_prep 2026-03-20 15:50:27.288312 | controller | - glance_deploy 2026-03-20 15:50:27.288316 | controller | - glance_deploy_cleanup 2026-03-20 15:50:27.288320 | controller | - ovn_prep 2026-03-20 15:50:27.288324 | controller | - ovn 2026-03-20 15:50:27.288328 | controller | - ovn_cleanup 2026-03-20 15:50:27.288332 | controller | - ovn_deploy_prep 2026-03-20 15:50:27.288337 | controller | - ovn_deploy 2026-03-20 15:50:27.288341 | controller | - ovn_deploy_cleanup 2026-03-20 15:50:27.288345 | controller | - neutron_prep 2026-03-20 15:50:27.288349 | controller | - neutron 2026-03-20 15:50:27.288353 | controller | - neutron_cleanup 2026-03-20 15:50:27.288357 | controller | - neutron_deploy_prep 2026-03-20 15:50:27.288361 | controller | - neutron_deploy 2026-03-20 15:50:27.288365 | controller | - neutron_deploy_cleanup 2026-03-20 15:50:27.288370 | controller | - cinder_prep 2026-03-20 15:50:27.288374 | controller | - cinder 2026-03-20 15:50:27.288378 | controller | - cinder_cleanup 2026-03-20 15:50:27.288382 | controller | - cinder_deploy_prep 2026-03-20 15:50:27.288386 | controller | - cinder_deploy 2026-03-20 15:50:27.288391 | controller | - cinder_deploy_cleanup 2026-03-20 15:50:27.288395 | controller | - rabbitmq_prep 2026-03-20 15:50:27.288399 | controller | - rabbitmq 2026-03-20 15:50:27.288403 | controller | - rabbitmq_cleanup 2026-03-20 15:50:27.288407 | controller | - rabbitmq_deploy_prep 2026-03-20 15:50:27.288411 | controller | - rabbitmq_deploy 2026-03-20 15:50:27.288416 | controller | - rabbitmq_deploy_cleanup 2026-03-20 15:50:27.288422 | controller | - ironic_prep 2026-03-20 15:50:27.288428 | controller | - ironic 2026-03-20 15:50:27.288435 | controller | - ironic_cleanup 2026-03-20 15:50:27.288442 | controller | - ironic_deploy_prep 2026-03-20 15:50:27.288447 | controller | - ironic_deploy 2026-03-20 15:50:27.288452 | controller | - ironic_deploy_cleanup 2026-03-20 15:50:27.288458 | controller | - octavia_prep 2026-03-20 15:50:27.288462 | controller | - octavia 2026-03-20 15:50:27.288466 | controller | - octavia_cleanup 2026-03-20 15:50:27.288471 | controller | - octavia_deploy_prep 2026-03-20 15:50:27.288475 | controller | - octavia_deploy 2026-03-20 15:50:27.288479 | controller | - octavia_deploy_cleanup 2026-03-20 15:50:27.288483 | controller | - designate_prep 2026-03-20 15:50:27.288487 | controller | - designate 2026-03-20 15:50:27.288492 | controller | - designate_cleanup 2026-03-20 15:50:27.288497 | controller | - designate_deploy_prep 2026-03-20 15:50:27.288506 | controller | - designate_deploy 2026-03-20 15:50:27.288512 | controller | - designate_deploy_cleanup 2026-03-20 15:50:27.288517 | controller | - nova_prep 2026-03-20 15:50:27.288527 | controller | - nova 2026-03-20 15:50:27.288533 | controller | - nova_cleanup 2026-03-20 15:50:27.288539 | controller | - nova_deploy_prep 2026-03-20 15:50:27.288545 | controller | - nova_deploy 2026-03-20 15:50:27.288550 | controller | - nova_deploy_cleanup 2026-03-20 15:50:27.288555 | controller | - mariadb_kuttl_run 2026-03-20 15:50:27.288559 | controller | - mariadb_kuttl 2026-03-20 15:50:27.288563 | controller | - kuttl_db_prep 2026-03-20 15:50:27.288567 | controller | - kuttl_db_cleanup 2026-03-20 15:50:27.288571 | controller | - kuttl_common_prep 2026-03-20 15:50:27.288580 | controller | - kuttl_common_cleanup 2026-03-20 15:50:27.288584 | controller | - keystone_kuttl_run 2026-03-20 15:50:27.288588 | controller | - keystone_kuttl 2026-03-20 15:50:27.288592 | controller | - barbican_kuttl_run 2026-03-20 15:50:27.288597 | controller | - barbican_kuttl 2026-03-20 15:50:27.288601 | controller | - placement_kuttl_run 2026-03-20 15:50:27.288605 | controller | - placement_kuttl 2026-03-20 15:50:27.288609 | controller | - cinder_kuttl_run 2026-03-20 15:50:27.288613 | controller | - cinder_kuttl 2026-03-20 15:50:27.288617 | controller | - neutron_kuttl_run 2026-03-20 15:50:27.288621 | controller | - neutron_kuttl 2026-03-20 15:50:27.288626 | controller | - octavia_kuttl_run 2026-03-20 15:50:27.288630 | controller | - octavia_kuttl 2026-03-20 15:50:27.288634 | controller | - designate_kuttl 2026-03-20 15:50:27.288667 | controller | - designate_kuttl_run 2026-03-20 15:50:27.288672 | controller | - ovn_kuttl_run 2026-03-20 15:50:27.288676 | controller | - ovn_kuttl 2026-03-20 15:50:27.288680 | controller | - infra_kuttl_run 2026-03-20 15:50:27.288684 | controller | - infra_kuttl 2026-03-20 15:50:27.288688 | controller | - ironic_kuttl_run 2026-03-20 15:50:27.288710 | controller | - ironic_kuttl 2026-03-20 15:50:27.288725 | controller | - ironic_kuttl_crc 2026-03-20 15:50:27.288730 | controller | - heat_kuttl_run 2026-03-20 15:50:27.288734 | controller | - heat_kuttl 2026-03-20 15:50:27.288738 | controller | - heat_kuttl_crc 2026-03-20 15:50:27.288742 | controller | - glance_kuttl_run 2026-03-20 15:50:27.288746 | controller | - glance_kuttl 2026-03-20 15:50:27.288751 | controller | - manila_kuttl_run 2026-03-20 15:50:27.288755 | controller | - manila_kuttl 2026-03-20 15:50:27.288759 | controller | - swift_kuttl_run 2026-03-20 15:50:27.288763 | controller | - swift_kuttl 2026-03-20 15:50:27.288767 | controller | - horizon_kuttl_run 2026-03-20 15:50:27.288771 | controller | - horizon_kuttl 2026-03-20 15:50:27.288775 | controller | - openstack_kuttl_prep 2026-03-20 15:50:27.288780 | controller | - openstack_kuttl_run 2026-03-20 15:50:27.288784 | controller | - openstack_kuttl_cleanup 2026-03-20 15:50:27.288788 | controller | - openstack_kuttl 2026-03-20 15:50:27.288792 | controller | - mariadb_chainsaw_run 2026-03-20 15:50:27.288796 | controller | - mariadb_chainsaw 2026-03-20 15:50:27.288800 | controller | - horizon_prep 2026-03-20 15:50:27.288804 | controller | - horizon 2026-03-20 15:50:27.288809 | controller | - horizon_cleanup 2026-03-20 15:50:27.288813 | controller | - horizon_deploy_prep 2026-03-20 15:50:27.288817 | controller | - horizon_deploy 2026-03-20 15:50:27.288821 | controller | - horizon_deploy_cleanup 2026-03-20 15:50:27.288825 | controller | - heat_prep 2026-03-20 15:50:27.288829 | controller | - heat 2026-03-20 15:50:27.288834 | controller | - heat_cleanup 2026-03-20 15:50:27.288838 | controller | - heat_deploy_prep 2026-03-20 15:50:27.288842 | controller | - heat_deploy 2026-03-20 15:50:27.288846 | controller | - heat_deploy_cleanup 2026-03-20 15:50:27.288850 | controller | - baremetal_prep 2026-03-20 15:50:27.288854 | controller | - baremetal 2026-03-20 15:50:27.288858 | controller | - baremetal_cleanup 2026-03-20 15:50:27.288862 | controller | - ceph_help 2026-03-20 15:50:27.288866 | controller | - ceph 2026-03-20 15:50:27.288871 | controller | - ceph_cleanup 2026-03-20 15:50:27.288875 | controller | - rook_prep 2026-03-20 15:50:27.288879 | controller | - rook 2026-03-20 15:50:27.288883 | controller | - rook_deploy_prep 2026-03-20 15:50:27.288887 | controller | - rook_deploy 2026-03-20 15:50:27.288891 | controller | - rook_crc_disk 2026-03-20 15:50:27.288895 | controller | - rook_cleanup 2026-03-20 15:50:27.288899 | controller | - lvms 2026-03-20 15:50:27.288904 | controller | - nmstate 2026-03-20 15:50:27.288908 | controller | - nncp 2026-03-20 15:50:27.288912 | controller | - nncp_cleanup 2026-03-20 15:50:27.288920 | controller | - netattach 2026-03-20 15:50:27.288924 | controller | - netattach_cleanup 2026-03-20 15:50:27.288928 | controller | - metallb 2026-03-20 15:50:27.288932 | controller | - metallb_config 2026-03-20 15:50:27.288936 | controller | - metallb_config_cleanup 2026-03-20 15:50:27.288940 | controller | - metallb_cleanup 2026-03-20 15:50:27.288945 | controller | - loki 2026-03-20 15:50:27.288949 | controller | - loki_cleanup 2026-03-20 15:50:27.288953 | controller | - loki_deploy 2026-03-20 15:50:27.288957 | controller | - loki_deploy_cleanup 2026-03-20 15:50:27.288961 | controller | - netobserv 2026-03-20 15:50:27.288965 | controller | - netobserv_cleanup 2026-03-20 15:50:27.288969 | controller | - netobserv_deploy 2026-03-20 15:50:27.288973 | controller | - netobserv_deploy_cleanup 2026-03-20 15:50:27.288978 | controller | - manila_prep 2026-03-20 15:50:27.288982 | controller | - manila 2026-03-20 15:50:27.288986 | controller | - manila_cleanup 2026-03-20 15:50:27.288990 | controller | - manila_deploy_prep 2026-03-20 15:50:27.288994 | controller | - manila_deploy 2026-03-20 15:50:27.288998 | controller | - manila_deploy_cleanup 2026-03-20 15:50:27.289002 | controller | - telemetry_prep 2026-03-20 15:50:27.289008 | controller | - telemetry 2026-03-20 15:50:27.289012 | controller | - telemetry_cleanup 2026-03-20 15:50:27.289016 | controller | - telemetry_deploy_prep 2026-03-20 15:50:27.289020 | controller | - telemetry_deploy 2026-03-20 15:50:27.289024 | controller | - telemetry_deploy_cleanup 2026-03-20 15:50:27.289028 | controller | - telemetry_kuttl_run 2026-03-20 15:50:27.289033 | controller | - telemetry_kuttl 2026-03-20 15:50:27.289037 | controller | - swift_prep 2026-03-20 15:50:27.289042 | controller | - swift 2026-03-20 15:50:27.289047 | controller | - swift_cleanup 2026-03-20 15:50:27.289053 | controller | - swift_deploy_prep 2026-03-20 15:50:27.289059 | controller | - swift_deploy 2026-03-20 15:50:27.289065 | controller | - swift_deploy_cleanup 2026-03-20 15:50:27.289069 | controller | - certmanager 2026-03-20 15:50:27.289073 | controller | - certmanager_cleanup 2026-03-20 15:50:27.289077 | controller | - validate_marketplace 2026-03-20 15:50:27.289082 | controller | - redis_deploy_prep 2026-03-20 15:50:27.289086 | controller | - redis_deploy 2026-03-20 15:50:27.289090 | controller | - redis_deploy_cleanup 2026-03-20 15:50:27.289094 | controller | - set_slower_etcd_profile 2026-03-20 15:50:27.289101 | controller | /home/zuul/src/github.com/openstack-k8s-operators/install_yamls/devsetup/Makefile: 2026-03-20 15:50:27.289107 | controller | - help 2026-03-20 15:50:27.289111 | controller | - download_tools 2026-03-20 15:50:27.289115 | controller | - nfs 2026-03-20 15:50:27.289119 | controller | - nfs_cleanup 2026-03-20 15:50:27.289123 | controller | - crc 2026-03-20 15:50:27.289127 | controller | - crc_cleanup 2026-03-20 15:50:27.289132 | controller | - crc_scrub 2026-03-20 15:50:27.289136 | controller | - crc_attach_default_interface 2026-03-20 15:50:27.289140 | controller | - crc_attach_default_interface_cleanup 2026-03-20 15:50:27.289146 | controller | - ipv6_lab_network 2026-03-20 15:50:27.289152 | controller | - ipv6_lab_network_cleanup 2026-03-20 15:50:27.289158 | controller | - ipv6_lab_nat64_router 2026-03-20 15:50:27.289164 | controller | - ipv6_lab_nat64_router_cleanup 2026-03-20 15:50:27.289170 | controller | - ipv6_lab_sno 2026-03-20 15:50:27.289176 | controller | - ipv6_lab_sno_cleanup 2026-03-20 15:50:27.289181 | controller | - ipv6_lab 2026-03-20 15:50:27.289187 | controller | - ipv6_lab_cleanup 2026-03-20 15:50:27.289193 | controller | - attach_default_interface 2026-03-20 15:50:27.289199 | controller | - attach_default_interface_cleanup 2026-03-20 15:50:27.289203 | controller | - network_isolation_bridge 2026-03-20 15:50:27.289208 | controller | - network_isolation_bridge_cleanup 2026-03-20 15:50:27.289212 | controller | - edpm_baremetal_compute 2026-03-20 15:50:27.289216 | controller | - edpm_compute 2026-03-20 15:50:27.289246 | controller | - edpm_compute_bootc 2026-03-20 15:50:27.289251 | controller | - edpm_ansible_runner 2026-03-20 15:50:27.289256 | controller | - edpm_computes_bgp 2026-03-20 15:50:27.289262 | controller | - edpm_compute_repos 2026-03-20 15:50:27.289277 | controller | - edpm_compute_cleanup 2026-03-20 15:50:27.289285 | controller | - edpm_networker 2026-03-20 15:50:27.289291 | controller | - edpm_networker_cleanup 2026-03-20 15:50:27.289302 | controller | - edpm_deploy_instance 2026-03-20 15:50:27.289309 | controller | - tripleo_deploy 2026-03-20 15:50:27.289314 | controller | - standalone_deploy 2026-03-20 15:50:27.289320 | controller | - standalone_sync 2026-03-20 15:50:27.289326 | controller | - standalone 2026-03-20 15:50:27.289331 | controller | - standalone_cleanup 2026-03-20 15:50:27.289337 | controller | - standalone_snapshot 2026-03-20 15:50:27.289342 | controller | - standalone_revert 2026-03-20 15:50:27.289348 | controller | - cifmw_prepare 2026-03-20 15:50:27.289354 | controller | - cifmw_cleanup 2026-03-20 15:50:27.289360 | controller | - bmaas_network 2026-03-20 15:50:27.289366 | controller | - bmaas_network_cleanup 2026-03-20 15:50:27.289371 | controller | - bmaas_route_crc_and_crc_bmaas_networks 2026-03-20 15:50:27.289377 | controller | - bmaas_route_crc_and_crc_bmaas_networks_cleanup 2026-03-20 15:50:27.289389 | controller | - bmaas_crc_attach_network 2026-03-20 15:50:27.297927 | controller | - bmaas_crc_attach_network_cleanup 2026-03-20 15:50:27.297963 | controller | - bmaas_crc_baremetal_bridge 2026-03-20 15:50:27.297968 | controller | - bmaas_crc_baremetal_bridge_cleanup 2026-03-20 15:50:27.297972 | controller | - bmaas_baremetal_net_nad 2026-03-20 15:50:27.297977 | controller | - bmaas_baremetal_net_nad_cleanup 2026-03-20 15:50:27.297981 | controller | - bmaas_metallb 2026-03-20 15:50:27.297985 | controller | - bmaas_metallb_cleanup 2026-03-20 15:50:27.297989 | controller | - bmaas_virtual_bms 2026-03-20 15:50:27.297994 | controller | - bmaas_virtual_bms_cleanup 2026-03-20 15:50:27.297998 | controller | - bmaas_sushy_emulator 2026-03-20 15:50:27.298002 | controller | - bmaas_sushy_emulator_cleanup 2026-03-20 15:50:27.298007 | controller | - bmaas_sushy_emulator_wait 2026-03-20 15:50:27.298011 | controller | - bmaas_generate_nodes_yaml 2026-03-20 15:50:27.298015 | controller | - bmaas 2026-03-20 15:50:27.298019 | controller | - bmaas_cleanup 2026-03-20 15:50:27.298023 | controller | failed: false 2026-03-20 15:50:27.298028 | controller | success: true 2026-03-20 15:50:27.298039 | controller | 2026-03-20 15:50:27.891341 | controller | TASK [install_yamls : Create the install_yamls parameters file dest={{ cifmw_basedir }}/artifacts/parameters/install-yamls-params.yml, content={{ 2026-03-20 15:50:27.891385 | controller | { 2026-03-20 15:50:27.891391 | controller | 'cifmw_install_yamls_environment': cifmw_install_yamls_environment, 2026-03-20 15:50:27.891397 | controller | 'cifmw_install_yamls_defaults': cifmw_install_yamls_defaults 2026-03-20 15:50:27.891401 | controller | } | to_nice_yaml 2026-03-20 15:50:27.891406 | controller | }}, mode=0644] *** 2026-03-20 15:50:27.891410 | controller | Friday 20 March 2026 15:50:27 +0000 (0:00:00.030) 0:01:02.921 ********** 2026-03-20 15:50:27.891415 | controller | Friday 20 March 2026 15:50:27 +0000 (0:00:00.030) 0:01:02.920 ********** 2026-03-20 15:50:27.891426 | controller | changed: [controller] 2026-03-20 15:50:27.919959 | controller | 2026-03-20 15:50:27.920014 | controller | TASK [install_yamls : Create empty cifmw_install_yamls_environment if needed cifmw_install_yamls_environment={}] *** 2026-03-20 15:50:27.920021 | controller | Friday 20 March 2026 15:50:27 +0000 (0:00:00.593) 0:01:03.515 ********** 2026-03-20 15:50:27.920026 | controller | Friday 20 March 2026 15:50:27 +0000 (0:00:00.593) 0:01:03.514 ********** 2026-03-20 15:50:27.920040 | controller | skipping: [controller] 2026-03-20 15:50:27.938523 | controller | 2026-03-20 15:50:27.938581 | controller | TASK [discover_latest_image : Set the discovered images dict to empty cifmw_discovered_images_dict={}] *** 2026-03-20 15:50:27.938588 | controller | Friday 20 March 2026 15:50:27 +0000 (0:00:00.028) 0:01:03.544 ********** 2026-03-20 15:50:27.938592 | controller | Friday 20 March 2026 15:50:27 +0000 (0:00:00.028) 0:01:03.543 ********** 2026-03-20 15:50:27.938605 | controller | ok: [controller] 2026-03-20 15:50:27.969592 | controller | 2026-03-20 15:50:27.969650 | controller | TASK [discover_latest_image : Discover images _raw_params=per-image.yml] ******* 2026-03-20 15:50:27.969656 | controller | Friday 20 March 2026 15:50:27 +0000 (0:00:00.018) 0:01:03.562 ********** 2026-03-20 15:50:27.969661 | controller | Friday 20 March 2026 15:50:27 +0000 (0:00:00.018) 0:01:03.561 ********** 2026-03-20 15:50:27.969675 | controller | included: /home/zuul/src/github.com/openstack-k8s-operators/ci-framework/roles/discover_latest_image/tasks/per-image.yml for controller => (item={'key': 'default', 'value': {'base_url': 'https://cloud.centos.org/centos/9-stream/x86_64/images/', 'qcow_prefix': 'CentOS-Stream-GenericCloud-', 'images_file': 'CHECKSUM'}}) 2026-03-20 15:50:28.001853 | controller | 2026-03-20 15:50:28.001902 | controller | TASK [discover_latest_image : Ensure the current image dict entry has the expected data that=['_expected_keys | difference(_given_keys) | length == 0'], fail_msg=The image discovery request dict's {{ _current_image.key }} entry does not match the expected schema. Requests must have the following keys: {{ _expected_keys }}, this entry had: {{ _given_keys }} See the discover_latest_image role's README for more information.] *** 2026-03-20 15:50:28.001913 | controller | Friday 20 March 2026 15:50:27 +0000 (0:00:00.030) 0:01:03.593 ********** 2026-03-20 15:50:28.001918 | controller | Friday 20 March 2026 15:50:27 +0000 (0:00:00.031) 0:01:03.592 ********** 2026-03-20 15:50:28.001929 | controller | ok: [controller] => 2026-03-20 15:50:28.455490 | controller | changed: false 2026-03-20 15:50:28.455549 | controller | msg: All assertions passed 2026-03-20 15:50:28.455555 | controller | 2026-03-20 15:50:28.455560 | controller | TASK [discover_latest_image : Get latest image for the current entry url={{ _current_image.value.base_url }}, image_prefix={{ _current_image.value.qcow_prefix }}, images_file={{ _current_image.value.images_file }}] *** 2026-03-20 15:50:28.455566 | controller | Friday 20 March 2026 15:50:28 +0000 (0:00:00.032) 0:01:03.626 ********** 2026-03-20 15:50:28.455571 | controller | Friday 20 March 2026 15:50:28 +0000 (0:00:00.032) 0:01:03.624 ********** 2026-03-20 15:50:28.455585 | controller | changed: [controller] 2026-03-20 15:50:28.481134 | controller | 2026-03-20 15:50:28.481167 | controller | TASK [discover_latest_image : Append the current return value to the discovered images dict cifmw_discovered_images_dict={{ cifmw_discovered_images_dict | combine({_current_image.key: discovered_image.data}) }}, cacheable=True] *** 2026-03-20 15:50:28.481174 | controller | Friday 20 March 2026 15:50:28 +0000 (0:00:00.453) 0:01:04.079 ********** 2026-03-20 15:50:28.481179 | controller | Friday 20 March 2026 15:50:28 +0000 (0:00:00.453) 0:01:04.078 ********** 2026-03-20 15:50:28.481188 | controller | ok: [controller] 2026-03-20 15:50:28.531908 | controller | 2026-03-20 15:50:28.531974 | controller | TASK [discover_latest_image : Export single-image facts if an image with name 'default' was discovered cifmw_discovered_image_name={{ cifmw_discovered_images_dict.default.image_name }}, cifmw_discovered_image_url={{ cifmw_discovered_images_dict.default.image_url }}, cifmw_discovered_hash={{ cifmw_discovered_images_dict.default.hash }}, cifmw_discovered_hash_algorithm={{ cifmw_discovered_images_dict.default.hash_algorithm }}, cacheable=True] *** 2026-03-20 15:50:28.531984 | controller | Friday 20 March 2026 15:50:28 +0000 (0:00:00.025) 0:01:04.105 ********** 2026-03-20 15:50:28.531991 | controller | Friday 20 March 2026 15:50:28 +0000 (0:00:00.025) 0:01:04.104 ********** 2026-03-20 15:50:28.532010 | controller | ok: [controller] 2026-03-20 15:50:29.198531 | controller | 2026-03-20 15:50:29.198574 | controller | TASK [cifmw_setup : Create artifacts with custom params mode=0644, dest={{ cifmw_basedir }}/artifacts/parameters/custom-params.yml, content={{ ci_framework_params | to_nice_yaml }}] *** 2026-03-20 15:50:29.198583 | controller | Friday 20 March 2026 15:50:28 +0000 (0:00:00.050) 0:01:04.155 ********** 2026-03-20 15:50:29.198590 | controller | Friday 20 March 2026 15:50:28 +0000 (0:00:00.050) 0:01:04.154 ********** 2026-03-20 15:50:29.198602 | controller | changed: [controller] 2026-03-20 15:50:29.231878 | controller | 2026-03-20 15:50:29.231913 | controller | PLAY [Install dev tools] ******************************************************* 2026-03-20 15:50:29.231919 | controller | 2026-03-20 15:50:29.231924 | controller | TASK [Assert that operator_name is set that=['operator_name is defined']] ****** 2026-03-20 15:50:29.231928 | controller | Friday 20 March 2026 15:50:29 +0000 (0:00:00.666) 0:01:04.822 ********** 2026-03-20 15:50:29.231933 | controller | Friday 20 March 2026 15:50:29 +0000 (0:00:00.666) 0:01:04.821 ********** 2026-03-20 15:50:29.231942 | controller | ok: [controller] => 2026-03-20 15:50:29.265893 | controller | changed: false 2026-03-20 15:50:29.265931 | controller | msg: All assertions passed 2026-03-20 15:50:29.265936 | controller | 2026-03-20 15:50:29.265941 | controller | TASK [Download install_yamls deps name=install_yamls_makes, tasks_from=make_download_tools] *** 2026-03-20 15:50:29.265947 | controller | Friday 20 March 2026 15:50:29 +0000 (0:00:00.033) 0:01:04.856 ********** 2026-03-20 15:50:29.265951 | controller | Friday 20 March 2026 15:50:29 +0000 (0:00:00.033) 0:01:04.855 ********** 2026-03-20 15:50:29.265960 | controller | 2026-03-20 15:50:29.289279 | controller | TASK [install_yamls_makes : Debug make_download_tools_env var=make_download_tools_env] *** 2026-03-20 15:50:29.289316 | controller | Friday 20 March 2026 15:50:29 +0000 (0:00:00.034) 0:01:04.890 ********** 2026-03-20 15:50:29.289328 | controller | Friday 20 March 2026 15:50:29 +0000 (0:00:00.034) 0:01:04.889 ********** 2026-03-20 15:50:29.289339 | controller | skipping: [controller] 2026-03-20 15:50:29.313800 | controller | 2026-03-20 15:50:29.313839 | controller | TASK [install_yamls_makes : Debug make_download_tools_params var=make_download_tools_params] *** 2026-03-20 15:50:29.313846 | controller | Friday 20 March 2026 15:50:29 +0000 (0:00:00.023) 0:01:04.913 ********** 2026-03-20 15:50:29.313850 | controller | Friday 20 March 2026 15:50:29 +0000 (0:00:00.023) 0:01:04.912 ********** 2026-03-20 15:50:29.313860 | controller | skipping: [controller] 2026-03-20 15:51:08.474802 | controller | 2026-03-20 15:51:08.474848 | controller | TASK [install_yamls_makes : Run download_tools output_dir={{ cifmw_basedir }}/artifacts, chdir=/home/zuul/src/github.com/openstack-k8s-operators/install_yamls/devsetup, script=make download_tools, dry_run={{ make_download_tools_dryrun|default(false)|bool }}, extra_args={{ dict((make_download_tools_env|default({})), **(make_download_tools_params|default({}))) }}] *** 2026-03-20 15:51:08.474855 | controller | Friday 20 March 2026 15:50:29 +0000 (0:00:00.024) 0:01:04.938 ********** 2026-03-20 15:51:08.474861 | controller | Friday 20 March 2026 15:50:29 +0000 (0:00:00.024) 0:01:04.936 ********** 2026-03-20 15:51:08.474871 | controller | [WARNING]: conditional statements should not include jinja2 templating 2026-03-20 15:51:08.491311 | controller | delimiters such as {{ }} or {% %}. Found: {{ make_download_tools_until | 2026-03-20 15:51:08.491391 | controller | default(true) }} 2026-03-20 15:51:08.491414 | controller | Follow script's output here: /home/zuul/ci-framework-data/logs/ci_script_000_run_download.log 2026-03-20 15:51:08.608788 | controller | changed: [controller] 2026-03-20 15:51:08.608863 | controller | 2026-03-20 15:51:08.608872 | controller | TASK [run_hook : Assert parameters are valid quiet=True, that=['_list_hooks is not string', '_list_hooks is not mapping', '_list_hooks is iterable', '(hooks | default([])) is not string', '(hooks | default([])) is not mapping', '(hooks | default([])) is iterable']] *** 2026-03-20 15:51:08.608880 | controller | Friday 20 March 2026 15:51:08 +0000 (0:00:39.177) 0:01:44.115 ********** 2026-03-20 15:51:08.608886 | controller | Friday 20 March 2026 15:51:08 +0000 (0:00:39.177) 0:01:44.114 ********** 2026-03-20 15:51:08.608904 | controller | ok: [controller] 2026-03-20 15:51:08.734600 | controller | 2026-03-20 15:51:08.734677 | controller | TASK [run_hook : Assert single hooks are all mappings quiet=True, that=['_not_mapping_hooks | length == 0'], msg=All single hooks must be a list of mappings or a mapping.] *** 2026-03-20 15:51:08.734687 | controller | Friday 20 March 2026 15:51:08 +0000 (0:00:00.116) 0:01:44.232 ********** 2026-03-20 15:51:08.734718 | controller | Friday 20 March 2026 15:51:08 +0000 (0:00:00.117) 0:01:44.231 ********** 2026-03-20 15:51:08.734740 | controller | ok: [controller] 2026-03-20 15:51:08.889234 | controller | 2026-03-20 15:51:08.889269 | controller | TASK [run_hook : Loop on hooks for pre_infra _raw_params={{ hook.type }}.yml] *** 2026-03-20 15:51:08.889275 | controller | Friday 20 March 2026 15:51:08 +0000 (0:00:00.125) 0:01:44.358 ********** 2026-03-20 15:51:08.889280 | controller | Friday 20 March 2026 15:51:08 +0000 (0:00:00.125) 0:01:44.357 ********** 2026-03-20 15:51:08.889290 | controller | included: /home/zuul/src/github.com/openstack-k8s-operators/ci-framework/roles/run_hook/tasks/playbook.yml for controller => (item={'name': 'Download needed tools', 'inventory': 'localhost,', 'connection': 'local', 'type': 'playbook', 'source': '/home/zuul/src/github.com/openstack-k8s-operators/install_yamls/devsetup/download_tools.yaml'}) 2026-03-20 15:51:08.963417 | controller | 2026-03-20 15:51:08.963453 | controller | TASK [run_hook : Set playbook path for Download needed tools cifmw_basedir={{ _bdir }}, hook_name={{ _hook_name }}, playbook_path={{ _play | realpath }}, log_path={{ _bdir }}/logs/{{ step }}_{{ _hook_name }}.log, extra_vars=-e namespace={{ cifmw_openstack_namespace }} {%- if hook.extra_vars is defined and hook.extra_vars|length > 0 -%} {% for key,value in hook.extra_vars.items() -%} {%- if key == 'file' %} -e "@{{ value }}" {%- else %} -e "{{ key }}={{ value }}" {%- endif %} {%- endfor %} {%- endif %}] *** 2026-03-20 15:51:08.963461 | controller | Friday 20 March 2026 15:51:08 +0000 (0:00:00.155) 0:01:44.513 ********** 2026-03-20 15:51:08.963467 | controller | Friday 20 March 2026 15:51:08 +0000 (0:00:00.155) 0:01:44.512 ********** 2026-03-20 15:51:08.963477 | controller | ok: [controller] 2026-03-20 15:51:09.196188 | controller | 2026-03-20 15:51:09.196226 | controller | TASK [run_hook : Get file stat path={{ playbook_path }}] *********************** 2026-03-20 15:51:09.196234 | controller | Friday 20 March 2026 15:51:08 +0000 (0:00:00.073) 0:01:44.587 ********** 2026-03-20 15:51:09.196240 | controller | Friday 20 March 2026 15:51:08 +0000 (0:00:00.074) 0:01:44.586 ********** 2026-03-20 15:51:09.196252 | controller | ok: [controller] 2026-03-20 15:51:09.218426 | controller | 2026-03-20 15:51:09.218469 | controller | TASK [run_hook : Fail if playbook doesn't exist msg=Playbook {{ playbook_path }} doesn't seem to exist.] *** 2026-03-20 15:51:09.218478 | controller | Friday 20 March 2026 15:51:09 +0000 (0:00:00.232) 0:01:44.820 ********** 2026-03-20 15:51:09.218484 | controller | Friday 20 March 2026 15:51:09 +0000 (0:00:00.232) 0:01:44.819 ********** 2026-03-20 15:51:09.218497 | controller | skipping: [controller] 2026-03-20 15:51:09.446537 | controller | 2026-03-20 15:51:09.446597 | controller | TASK [run_hook : Get parameters files paths={{ 2026-03-20 15:51:09.446611 | controller | (cifmw_basedir, 'artifacts/parameters') | path_join 2026-03-20 15:51:09.446622 | controller | }}, file_type=file, patterns=*.yml] *** 2026-03-20 15:51:09.446632 | controller | Friday 20 March 2026 15:51:09 +0000 (0:00:00.022) 0:01:44.842 ********** 2026-03-20 15:51:09.446643 | controller | Friday 20 March 2026 15:51:09 +0000 (0:00:00.022) 0:01:44.841 ********** 2026-03-20 15:51:09.446662 | controller | ok: [controller] 2026-03-20 15:51:09.487067 | controller | 2026-03-20 15:51:09.487119 | controller | TASK [run_hook : Add parameters artifacts as extra variables extra_vars={{ extra_vars }} {% for file in cifmw_run_hook_parameters_files.files %} -e "@{{ file.path }}" {%- endfor %}] *** 2026-03-20 15:51:09.487132 | controller | Friday 20 March 2026 15:51:09 +0000 (0:00:00.228) 0:01:45.070 ********** 2026-03-20 15:51:09.487143 | controller | Friday 20 March 2026 15:51:09 +0000 (0:00:00.228) 0:01:45.069 ********** 2026-03-20 15:51:09.487161 | controller | ok: [controller] 2026-03-20 15:51:09.725205 | controller | 2026-03-20 15:51:09.725260 | controller | TASK [run_hook : Ensure log directory exists path={{ log_path | dirname }}, state=directory, mode=0755] *** 2026-03-20 15:51:09.725273 | controller | Friday 20 March 2026 15:51:09 +0000 (0:00:00.039) 0:01:45.110 ********** 2026-03-20 15:51:09.725283 | controller | Friday 20 March 2026 15:51:09 +0000 (0:00:00.040) 0:01:45.109 ********** 2026-03-20 15:51:09.725301 | controller | ok: [controller] 2026-03-20 15:51:09.943790 | controller | 2026-03-20 15:51:09.943831 | controller | TASK [run_hook : Ensure artifacts directory exists path={{ cifmw_basedir }}/artifacts, state=directory, mode=0755] *** 2026-03-20 15:51:09.943839 | controller | Friday 20 March 2026 15:51:09 +0000 (0:00:00.238) 0:01:45.348 ********** 2026-03-20 15:51:09.943846 | controller | Friday 20 March 2026 15:51:09 +0000 (0:00:00.238) 0:01:45.347 ********** 2026-03-20 15:51:09.943858 | controller | ok: [controller] 2026-03-20 15:51:57.629682 | controller | 2026-03-20 15:51:57.629744 | controller | TASK [run_hook : Run hook without retry - Download needed tools] *************** 2026-03-20 15:51:57.629753 | controller | Friday 20 March 2026 15:51:09 +0000 (0:00:00.219) 0:01:45.568 ********** 2026-03-20 15:51:57.629759 | controller | Friday 20 March 2026 15:51:09 +0000 (0:00:00.219) 0:01:45.566 ********** 2026-03-20 15:51:57.629775 | controller | Follow script's output here: /home/zuul/ci-framework-data/logs/ci_script_001_run_hook_without_retry.log 2026-03-20 15:51:57.653089 | controller | changed: [controller] 2026-03-20 15:51:57.653119 | controller | 2026-03-20 15:51:57.653125 | controller | TASK [run_hook : Run hook with retry - Download needed tools] ****************** 2026-03-20 15:51:57.653131 | controller | Friday 20 March 2026 15:51:57 +0000 (0:00:47.685) 0:02:33.253 ********** 2026-03-20 15:51:57.653135 | controller | Friday 20 March 2026 15:51:57 +0000 (0:00:47.685) 0:02:33.252 ********** 2026-03-20 15:51:57.653143 | controller | skipping: [controller] 2026-03-20 15:51:57.850405 | controller | 2026-03-20 15:51:57.850442 | controller | TASK [run_hook : Check if we have a file path={{ cifmw_basedir }}/artifacts/{{ step }}_{{ hook_name }}.yml] *** 2026-03-20 15:51:57.850451 | controller | Friday 20 March 2026 15:51:57 +0000 (0:00:00.023) 0:02:33.277 ********** 2026-03-20 15:51:57.850457 | controller | Friday 20 March 2026 15:51:57 +0000 (0:00:00.023) 0:02:33.276 ********** 2026-03-20 15:51:57.850469 | controller | ok: [controller] 2026-03-20 15:51:57.921536 | controller | 2026-03-20 15:51:57.921566 | controller | TASK [run_hook : Load generated content in main playbook file={{ cifmw_basedir }}/artifacts/{{ step }}_{{ hook_name }}.yml] *** 2026-03-20 15:51:57.921573 | controller | Friday 20 March 2026 15:51:57 +0000 (0:00:00.197) 0:02:33.474 ********** 2026-03-20 15:51:57.921577 | controller | Friday 20 March 2026 15:51:57 +0000 (0:00:00.197) 0:02:33.473 ********** 2026-03-20 15:51:57.921586 | controller | skipping: [controller] 2026-03-20 15:51:58.013009 | controller | 2026-03-20 15:51:58.013043 | controller | PLAY [Prepare host virtualization] ********************************************* 2026-03-20 15:51:58.013050 | controller | 2026-03-20 15:51:58.013054 | controller | TASK [cifmw_setup : Load parameters files dir={{ cifmw_basedir }}/artifacts/parameters] *** 2026-03-20 15:51:58.013059 | controller | Friday 20 March 2026 15:51:57 +0000 (0:00:00.071) 0:02:33.546 ********** 2026-03-20 15:51:58.013063 | controller | Friday 20 March 2026 15:51:57 +0000 (0:00:00.071) 0:02:33.544 ********** 2026-03-20 15:51:58.013072 | controller | ok: [controller] 2026-03-20 15:51:58.037654 | controller | 2026-03-20 15:51:58.037686 | controller | TASK [Ensure libvirt is present/configured name=libvirt_manager] *************** 2026-03-20 15:51:58.037708 | controller | Friday 20 March 2026 15:51:58 +0000 (0:00:00.091) 0:02:33.637 ********** 2026-03-20 15:51:58.037714 | controller | Friday 20 March 2026 15:51:58 +0000 (0:00:00.091) 0:02:33.636 ********** 2026-03-20 15:51:58.037723 | controller | skipping: [controller] 2026-03-20 15:51:58.106373 | controller | 2026-03-20 15:51:58.106404 | controller | TASK [Perpare OpenShift provisioner node name=openshift_provisioner_node] ****** 2026-03-20 15:51:58.106411 | controller | Friday 20 March 2026 15:51:58 +0000 (0:00:00.024) 0:02:33.662 ********** 2026-03-20 15:51:58.106416 | controller | Friday 20 March 2026 15:51:58 +0000 (0:00:00.024) 0:02:33.660 ********** 2026-03-20 15:51:58.106426 | controller | skipping: [controller] 2026-03-20 15:51:58.195884 | controller | 2026-03-20 15:51:58.195956 | controller | PLAY [Build dataset hook] ****************************************************** 2026-03-20 15:51:58.195966 | controller | 2026-03-20 15:51:58.195972 | controller | TASK [cifmw_setup : Load parameters files dir={{ cifmw_basedir }}/artifacts/parameters] *** 2026-03-20 15:51:58.195978 | controller | Friday 20 March 2026 15:51:58 +0000 (0:00:00.068) 0:02:33.730 ********** 2026-03-20 15:51:58.195984 | controller | Friday 20 March 2026 15:51:58 +0000 (0:00:00.068) 0:02:33.729 ********** 2026-03-20 15:51:58.196003 | controller | ok: [controller] 2026-03-20 15:51:58.394171 | controller | 2026-03-20 15:51:58.394202 | controller | TASK [networking_mapper : Check for Networking Environment Definition file existence path={{ cifmw_networking_mapper_networking_env_def_path }}] *** 2026-03-20 15:51:58.394208 | controller | Friday 20 March 2026 15:51:58 +0000 (0:00:00.089) 0:02:33.820 ********** 2026-03-20 15:51:58.394214 | controller | Friday 20 March 2026 15:51:58 +0000 (0:00:00.089) 0:02:33.819 ********** 2026-03-20 15:51:58.394222 | controller | ok: [controller] 2026-03-20 15:51:58.424014 | controller | 2026-03-20 15:51:58.424053 | controller | TASK [networking_mapper : Check for Networking Definition file existance that=['_net_env_def_stat.stat.exists'], msg=Ensure that the Networking Environment Definition file exists in {{ cifmw_networking_mapper_networking_env_def_path }}, quiet=True] *** 2026-03-20 15:51:58.424066 | controller | Friday 20 March 2026 15:51:58 +0000 (0:00:00.198) 0:02:34.018 ********** 2026-03-20 15:51:58.424073 | controller | Friday 20 March 2026 15:51:58 +0000 (0:00:00.198) 0:02:34.017 ********** 2026-03-20 15:51:58.424084 | controller | skipping: [controller] 2026-03-20 15:51:58.460992 | controller | 2026-03-20 15:51:58.461053 | controller | TASK [networking_mapper : Load the Networking Definition from file path={{ cifmw_networking_mapper_networking_env_def_path }}] *** 2026-03-20 15:51:58.461059 | controller | Friday 20 March 2026 15:51:58 +0000 (0:00:00.029) 0:02:34.047 ********** 2026-03-20 15:51:58.461064 | controller | Friday 20 March 2026 15:51:58 +0000 (0:00:00.029) 0:02:34.046 ********** 2026-03-20 15:51:58.461103 | controller | skipping: [controller] 2026-03-20 15:51:58.495137 | controller | 2026-03-20 15:51:58.495214 | controller | TASK [networking_mapper : Set cifmw_networking_env_definition is present cifmw_networking_env_definition={{ 2026-03-20 15:51:58.495223 | controller | _net_env_def_slurp['content'] | 2026-03-20 15:51:58.495230 | controller | b64decode | 2026-03-20 15:51:58.495237 | controller | from_yaml 2026-03-20 15:51:58.495244 | controller | }}, cacheable=True] *** 2026-03-20 15:51:58.495251 | controller | Friday 20 March 2026 15:51:58 +0000 (0:00:00.037) 0:02:34.085 ********** 2026-03-20 15:51:58.495257 | controller | Friday 20 March 2026 15:51:58 +0000 (0:00:00.037) 0:02:34.083 ********** 2026-03-20 15:51:58.495277 | controller | skipping: [controller] 2026-03-20 15:51:58.518070 | controller | 2026-03-20 15:51:58.518158 | controller | TASK [Deploy OCP using Hive name=hive] ***************************************** 2026-03-20 15:51:58.518166 | controller | Friday 20 March 2026 15:51:58 +0000 (0:00:00.033) 0:02:34.119 ********** 2026-03-20 15:51:58.518173 | controller | Friday 20 March 2026 15:51:58 +0000 (0:00:00.033) 0:02:34.117 ********** 2026-03-20 15:51:58.518182 | controller | skipping: [controller] 2026-03-20 15:51:58.548169 | controller | 2026-03-20 15:51:58.548201 | controller | TASK [Prepare CRC name=rhol_crc] *********************************************** 2026-03-20 15:51:58.548207 | controller | Friday 20 March 2026 15:51:58 +0000 (0:00:00.023) 0:02:34.142 ********** 2026-03-20 15:51:58.548212 | controller | Friday 20 March 2026 15:51:58 +0000 (0:00:00.023) 0:02:34.141 ********** 2026-03-20 15:51:58.548221 | controller | skipping: [controller] 2026-03-20 15:51:58.577595 | controller | 2026-03-20 15:51:58.577628 | controller | TASK [Deploy OpenShift cluster using dev-scripts name=devscripts] ************** 2026-03-20 15:51:58.577634 | controller | Friday 20 March 2026 15:51:58 +0000 (0:00:00.030) 0:02:34.172 ********** 2026-03-20 15:51:58.577639 | controller | Friday 20 March 2026 15:51:58 +0000 (0:00:00.030) 0:02:34.171 ********** 2026-03-20 15:51:58.577648 | controller | skipping: [controller] 2026-03-20 15:51:58.777999 | controller | 2026-03-20 15:51:58.778077 | controller | TASK [openshift_login : Ensure output directory exists path={{ cifmw_openshift_login_basedir }}/artifacts, state=directory, mode=0755] *** 2026-03-20 15:51:58.778091 | controller | Friday 20 March 2026 15:51:58 +0000 (0:00:00.029) 0:02:34.201 ********** 2026-03-20 15:51:58.778102 | controller | Friday 20 March 2026 15:51:58 +0000 (0:00:00.029) 0:02:34.200 ********** 2026-03-20 15:51:58.778125 | controller | ok: [controller] 2026-03-20 15:51:58.816438 | controller | 2026-03-20 15:51:58.816540 | controller | TASK [openshift_login : OpenShift login _raw_params=login.yml] ***************** 2026-03-20 15:51:58.816557 | controller | Friday 20 March 2026 15:51:58 +0000 (0:00:00.200) 0:02:34.402 ********** 2026-03-20 15:51:58.816571 | controller | Friday 20 March 2026 15:51:58 +0000 (0:00:00.200) 0:02:34.401 ********** 2026-03-20 15:51:58.816601 | controller | included: /home/zuul/src/github.com/openstack-k8s-operators/ci-framework/roles/openshift_login/tasks/login.yml for controller 2026-03-20 15:51:58.861605 | controller | 2026-03-20 15:51:58.861651 | controller | TASK [openshift_login : Check if the password file is present path={{ cifmw_openshift_login_password_file | default(cifmw_openshift_password_file) }}] *** 2026-03-20 15:51:58.861664 | controller | Friday 20 March 2026 15:51:58 +0000 (0:00:00.038) 0:02:34.440 ********** 2026-03-20 15:51:58.861674 | controller | Friday 20 March 2026 15:51:58 +0000 (0:00:00.038) 0:02:34.439 ********** 2026-03-20 15:51:58.861716 | controller | skipping: [controller] 2026-03-20 15:51:58.891367 | controller | 2026-03-20 15:51:58.891404 | controller | TASK [openshift_login : Fetch user password content src={{ cifmw_openshift_login_password_file | default(cifmw_openshift_password_file) }}] *** 2026-03-20 15:51:58.891414 | controller | Friday 20 March 2026 15:51:58 +0000 (0:00:00.044) 0:02:34.485 ********** 2026-03-20 15:51:58.891421 | controller | Friday 20 March 2026 15:51:58 +0000 (0:00:00.045) 0:02:34.484 ********** 2026-03-20 15:51:58.891433 | controller | skipping: [controller] 2026-03-20 15:51:58.919030 | controller | 2026-03-20 15:51:58.919072 | controller | TASK [openshift_login : Set user password as a fact cifmw_openshift_login_password={{ cifmw_openshift_login_password_file_slurp.content | b64decode }}, cacheable=True] *** 2026-03-20 15:51:58.919085 | controller | Friday 20 March 2026 15:51:58 +0000 (0:00:00.029) 0:02:34.515 ********** 2026-03-20 15:51:58.919095 | controller | Friday 20 March 2026 15:51:58 +0000 (0:00:00.029) 0:02:34.514 ********** 2026-03-20 15:51:58.919111 | controller | skipping: [controller] 2026-03-20 15:51:58.960959 | controller | 2026-03-20 15:51:58.960998 | controller | TASK [openshift_login : Set role variables cifmw_openshift_login_kubeconfig={{ 2026-03-20 15:51:58.961007 | controller | cifmw_openshift_login_kubeconfig | 2026-03-20 15:51:58.961014 | controller | default(cifmw_openshift_kubeconfig) | 2026-03-20 15:51:58.961021 | controller | default( 2026-03-20 15:51:58.961028 | controller | ansible_env.KUBECONFIG if 'KUBECONFIG' in ansible_env else 2026-03-20 15:51:58.961035 | controller | cifmw_openshift_login_kubeconfig_default_path 2026-03-20 15:51:58.961041 | controller | ) | trim 2026-03-20 15:51:58.961049 | controller | }}, cifmw_openshift_login_user={{ cifmw_openshift_login_user | default(cifmw_openshift_user) | default(omit) }}, cifmw_openshift_login_password={{ cifmw_openshift_login_password | default(cifmw_openshift_password) | default(omit) }}, cifmw_openshift_login_api={{ cifmw_openshift_login_api | default(cifmw_openshift_api) | default(omit) }}, cifmw_openshift_login_cert_login={{ cifmw_openshift_login_cert_login | default(false)}}, cifmw_openshift_login_provided_token={{ cifmw_openshift_provided_token | default(omit) }}, cacheable=True] *** 2026-03-20 15:51:58.961057 | controller | Friday 20 March 2026 15:51:58 +0000 (0:00:00.027) 0:02:34.543 ********** 2026-03-20 15:51:58.961064 | controller | Friday 20 March 2026 15:51:58 +0000 (0:00:00.027) 0:02:34.541 ********** 2026-03-20 15:51:58.961076 | controller | ok: [controller] 2026-03-20 15:51:59.161273 | controller | 2026-03-20 15:51:59.161306 | controller | TASK [openshift_login : Check if kubeconfig exists path={{ cifmw_openshift_login_kubeconfig }}] *** 2026-03-20 15:51:59.161312 | controller | Friday 20 March 2026 15:51:58 +0000 (0:00:00.041) 0:02:34.585 ********** 2026-03-20 15:51:59.161317 | controller | Friday 20 March 2026 15:51:58 +0000 (0:00:00.041) 0:02:34.583 ********** 2026-03-20 15:51:59.161326 | controller | ok: [controller] 2026-03-20 15:51:59.190181 | controller | 2026-03-20 15:51:59.190213 | controller | TASK [openshift_login : Assert that enough data is provided to log in to OpenShift that=cifmw_openshift_login_kubeconfig_stat.stat.exists or (cifmw_openshift_login_provided_token is defined and cifmw_openshift_login_provided_token != '') or ( 2026-03-20 15:51:59.190220 | controller | (cifmw_openshift_login_user is defined) and 2026-03-20 15:51:59.190226 | controller | (cifmw_openshift_login_password is defined) and 2026-03-20 15:51:59.190230 | controller | (cifmw_openshift_login_api is defined) 2026-03-20 15:51:59.190235 | controller | ), msg=If an existing kubeconfig is not provided user/pwd or provided/initial token and API URL must be given] *** 2026-03-20 15:51:59.190239 | controller | Friday 20 March 2026 15:51:59 +0000 (0:00:00.200) 0:02:34.785 ********** 2026-03-20 15:51:59.190244 | controller | Friday 20 March 2026 15:51:59 +0000 (0:00:00.200) 0:02:34.784 ********** 2026-03-20 15:51:59.190253 | controller | ok: [controller] => 2026-03-20 15:51:59.213582 | controller | changed: false 2026-03-20 15:51:59.213614 | controller | msg: All assertions passed 2026-03-20 15:51:59.213620 | controller | 2026-03-20 15:51:59.213625 | controller | TASK [openshift_login : Fetch kubeconfig content src={{ cifmw_openshift_login_kubeconfig }}] *** 2026-03-20 15:51:59.213631 | controller | Friday 20 March 2026 15:51:59 +0000 (0:00:00.028) 0:02:34.814 ********** 2026-03-20 15:51:59.213639 | controller | Friday 20 March 2026 15:51:59 +0000 (0:00:00.028) 0:02:34.813 ********** 2026-03-20 15:51:59.213652 | controller | skipping: [controller] 2026-03-20 15:51:59.236759 | controller | 2026-03-20 15:51:59.236797 | controller | TASK [openshift_login : Fetch x509 key based users cifmw_openshift_login_key_based_users={{ 2026-03-20 15:51:59.236804 | controller | ( 2026-03-20 15:51:59.236809 | controller | cifmw_openshift_login_kubeconfig_content_b64.content | 2026-03-20 15:51:59.236814 | controller | b64decode | 2026-03-20 15:51:59.236818 | controller | from_yaml 2026-03-20 15:51:59.236823 | controller | ). users | default([]) | 2026-03-20 15:51:59.236830 | controller | selectattr('user.client-certificate-data', 'defined') | 2026-03-20 15:51:59.236835 | controller | map(attribute="name") | 2026-03-20 15:51:59.236839 | controller | map("split", "/") | 2026-03-20 15:51:59.236844 | controller | map("first") 2026-03-20 15:51:59.236848 | controller | }}, cacheable=True] *** 2026-03-20 15:51:59.236852 | controller | Friday 20 March 2026 15:51:59 +0000 (0:00:00.023) 0:02:34.837 ********** 2026-03-20 15:51:59.236857 | controller | Friday 20 March 2026 15:51:59 +0000 (0:00:00.023) 0:02:34.836 ********** 2026-03-20 15:51:59.236866 | controller | skipping: [controller] 2026-03-20 15:51:59.267739 | controller | 2026-03-20 15:51:59.267777 | controller | TASK [openshift_login : Assign key based user if not provided and available cifmw_openshift_login_user={{ 2026-03-20 15:51:59.267788 | controller | (cifmw_openshift_login_assume_cert_system_user | ternary('system:', '')) + 2026-03-20 15:51:59.267798 | controller | (cifmw_openshift_login_key_based_users | map('replace', 'system:', '') | unique | first) 2026-03-20 15:51:59.267808 | controller | }}, cifmw_openshift_login_cert_login=True, cacheable=True] *** 2026-03-20 15:51:59.267817 | controller | Friday 20 March 2026 15:51:59 +0000 (0:00:00.023) 0:02:34.861 ********** 2026-03-20 15:51:59.267830 | controller | Friday 20 March 2026 15:51:59 +0000 (0:00:00.023) 0:02:34.859 ********** 2026-03-20 15:51:59.267846 | controller | skipping: [controller] 2026-03-20 15:51:59.301444 | controller | 2026-03-20 15:51:59.301485 | controller | TASK [openshift_login : Set the retry count cifmw_openshift_login_retries_cnt={{ 2026-03-20 15:51:59.301494 | controller | 0 if cifmw_openshift_login_retries_cnt is undefined else 2026-03-20 15:51:59.301502 | controller | cifmw_openshift_login_retries_cnt|int + 1 2026-03-20 15:51:59.301509 | controller | }}] *** 2026-03-20 15:51:59.301516 | controller | Friday 20 March 2026 15:51:59 +0000 (0:00:00.030) 0:02:34.891 ********** 2026-03-20 15:51:59.301523 | controller | Friday 20 March 2026 15:51:59 +0000 (0:00:00.030) 0:02:34.890 ********** 2026-03-20 15:51:59.301536 | controller | ok: [controller] 2026-03-20 15:51:59.340902 | controller | 2026-03-20 15:51:59.340963 | controller | TASK [openshift_login : Fetch token _raw_params=try_login.yml] ***************** 2026-03-20 15:51:59.340970 | controller | Friday 20 March 2026 15:51:59 +0000 (0:00:00.033) 0:02:34.925 ********** 2026-03-20 15:51:59.340974 | controller | Friday 20 March 2026 15:51:59 +0000 (0:00:00.033) 0:02:34.924 ********** 2026-03-20 15:51:59.340989 | controller | included: /home/zuul/src/github.com/openstack-k8s-operators/ci-framework/roles/openshift_login/tasks/try_login.yml for controller 2026-03-20 15:51:59.365608 | controller | 2026-03-20 15:51:59.365662 | controller | TASK [openshift_login : Try get OpenShift access token _raw_params=oc whoami -t] *** 2026-03-20 15:51:59.365669 | controller | Friday 20 March 2026 15:51:59 +0000 (0:00:00.039) 0:02:34.965 ********** 2026-03-20 15:51:59.365673 | controller | Friday 20 March 2026 15:51:59 +0000 (0:00:00.039) 0:02:34.963 ********** 2026-03-20 15:51:59.365687 | controller | skipping: [controller] 2026-03-20 15:52:00.003256 | controller | 2026-03-20 15:52:00.003297 | controller | TASK [openshift_login : Fetch OpenShift token output_dir={{ cifmw_openshift_login_basedir }}/artifacts, script=oc login {%- if cifmw_openshift_login_provided_token is not defined %} {%- if cifmw_openshift_login_user is defined %} -u {{ cifmw_openshift_login_user }} {%- endif %} {%- if cifmw_openshift_login_password is defined %} -p {{ cifmw_openshift_login_password }} {%- endif %} {% else %} --token={{ cifmw_openshift_login_provided_token }} {%- endif %} {%- if cifmw_openshift_login_skip_tls_verify|bool %} --insecure-skip-tls-verify=true {%- endif %} {%- if cifmw_openshift_login_api is defined %} {{ cifmw_openshift_login_api }} {%- endif %}] *** 2026-03-20 15:52:00.003313 | controller | Friday 20 March 2026 15:51:59 +0000 (0:00:00.024) 0:02:34.989 ********** 2026-03-20 15:52:00.003321 | controller | Friday 20 March 2026 15:51:59 +0000 (0:00:00.024) 0:02:34.988 ********** 2026-03-20 15:52:00.003333 | controller | Follow script's output here: /home/zuul/ci-framework-data/logs/ci_script_002_fetch_openshift.log 2026-03-20 15:52:00.034632 | controller | changed: [controller] 2026-03-20 15:52:00.034671 | controller | 2026-03-20 15:52:00.034680 | controller | TASK [openshift_login : Ensure kubeconfig is provided that=cifmw_openshift_login_kubeconfig != ""] *** 2026-03-20 15:52:00.034688 | controller | Friday 20 March 2026 15:52:00 +0000 (0:00:00.637) 0:02:35.627 ********** 2026-03-20 15:52:00.034727 | controller | Friday 20 March 2026 15:52:00 +0000 (0:00:00.637) 0:02:35.626 ********** 2026-03-20 15:52:00.034744 | controller | ok: [controller] => 2026-03-20 15:52:00.349393 | controller | changed: false 2026-03-20 15:52:00.349423 | controller | msg: All assertions passed 2026-03-20 15:52:00.349429 | controller | 2026-03-20 15:52:00.349434 | controller | TASK [openshift_login : Fetch new OpenShift access token _raw_params=oc whoami -t] *** 2026-03-20 15:52:00.349439 | controller | Friday 20 March 2026 15:52:00 +0000 (0:00:00.031) 0:02:35.658 ********** 2026-03-20 15:52:00.349443 | controller | Friday 20 March 2026 15:52:00 +0000 (0:00:00.031) 0:02:35.657 ********** 2026-03-20 15:52:00.349452 | controller | changed: [controller] 2026-03-20 15:52:00.375307 | controller | 2026-03-20 15:52:00.375338 | controller | TASK [openshift_login : Set new OpenShift token cifmw_openshift_login_token={{ 2026-03-20 15:52:00.375344 | controller | (not cifmw_openshift_login_new_token_out.skipped | default(false)) | 2026-03-20 15:52:00.375349 | controller | ternary(cifmw_openshift_login_new_token_out.stdout, cifmw_openshift_login_whoami_out.stdout) 2026-03-20 15:52:00.375354 | controller | }}, cacheable=True] *** 2026-03-20 15:52:00.375359 | controller | Friday 20 March 2026 15:52:00 +0000 (0:00:00.315) 0:02:35.973 ********** 2026-03-20 15:52:00.375363 | controller | Friday 20 March 2026 15:52:00 +0000 (0:00:00.314) 0:02:35.972 ********** 2026-03-20 15:52:00.375373 | controller | ok: [controller] 2026-03-20 15:52:00.650514 | controller | 2026-03-20 15:52:00.650659 | controller | TASK [openshift_login : Fetch OpenShift API URL _raw_params=oc whoami --show-server=true] *** 2026-03-20 15:52:00.650672 | controller | Friday 20 March 2026 15:52:00 +0000 (0:00:00.025) 0:02:35.999 ********** 2026-03-20 15:52:00.650681 | controller | Friday 20 March 2026 15:52:00 +0000 (0:00:00.025) 0:02:35.998 ********** 2026-03-20 15:52:00.650714 | controller | changed: [controller] 2026-03-20 15:52:00.949545 | controller | 2026-03-20 15:52:00.949578 | controller | TASK [openshift_login : Fetch OpenShift kubeconfig context _raw_params=oc whoami -c] *** 2026-03-20 15:52:00.949584 | controller | Friday 20 March 2026 15:52:00 +0000 (0:00:00.274) 0:02:36.274 ********** 2026-03-20 15:52:00.949589 | controller | Friday 20 March 2026 15:52:00 +0000 (0:00:00.274) 0:02:36.273 ********** 2026-03-20 15:52:00.949597 | controller | changed: [controller] 2026-03-20 15:52:01.266439 | controller | 2026-03-20 15:52:01.266473 | controller | TASK [openshift_login : Fetch OpenShift current user _raw_params=oc whoami] **** 2026-03-20 15:52:01.266480 | controller | Friday 20 March 2026 15:52:00 +0000 (0:00:00.299) 0:02:36.573 ********** 2026-03-20 15:52:01.266484 | controller | Friday 20 March 2026 15:52:00 +0000 (0:00:00.299) 0:02:36.572 ********** 2026-03-20 15:52:01.266494 | controller | changed: [controller] 2026-03-20 15:52:01.303510 | controller | 2026-03-20 15:52:01.303544 | controller | TASK [openshift_login : Set OpenShift user, context and API facts cifmw_openshift_login_api={{ cifmw_openshift_login_api_out.stdout }}, cifmw_openshift_login_context={{ cifmw_openshift_login_context_out.stdout }}, cifmw_openshift_login_user={{ _oauth_user }}, cifmw_openshift_kubeconfig={{ cifmw_openshift_login_kubeconfig }}, cifmw_openshift_api={{ cifmw_openshift_login_api_out.stdout }}, cifmw_openshift_context={{ cifmw_openshift_login_context_out.stdout }}, cifmw_openshift_user={{ _oauth_user }}, cifmw_openshift_token={{ cifmw_openshift_login_token | default(omit) }}, cifmw_install_yamls_environment={{ ( cifmw_install_yamls_environment | 2026-03-20 15:52:01.303556 | controller | combine({'KUBECONFIG': cifmw_openshift_login_kubeconfig}) 2026-03-20 15:52:01.303562 | controller | ) if cifmw_install_yamls_environment is defined else omit 2026-03-20 15:52:01.303567 | controller | }}, cacheable=True] *** 2026-03-20 15:52:01.303572 | controller | Friday 20 March 2026 15:52:01 +0000 (0:00:00.316) 0:02:36.890 ********** 2026-03-20 15:52:01.303577 | controller | Friday 20 March 2026 15:52:01 +0000 (0:00:00.316) 0:02:36.889 ********** 2026-03-20 15:52:01.303589 | controller | ok: [controller] 2026-03-20 15:52:01.867166 | controller | 2026-03-20 15:52:01.867228 | controller | TASK [openshift_login : Create the openshift_login parameters file dest={{ cifmw_basedir }}/artifacts/parameters/openshift-login-params.yml, content={{ cifmw_openshift_login_params_content | from_yaml | to_nice_yaml }}, mode=0600] *** 2026-03-20 15:52:01.867248 | controller | Friday 20 March 2026 15:52:01 +0000 (0:00:00.037) 0:02:36.927 ********** 2026-03-20 15:52:01.867261 | controller | Friday 20 March 2026 15:52:01 +0000 (0:00:00.037) 0:02:36.926 ********** 2026-03-20 15:52:01.867281 | controller | changed: [controller] 2026-03-20 15:52:02.214600 | controller | 2026-03-20 15:52:02.214666 | controller | TASK [openshift_login : Read the install yamls parameters file path={{ cifmw_basedir }}/artifacts/parameters/install-yamls-params.yml] *** 2026-03-20 15:52:02.214673 | controller | Friday 20 March 2026 15:52:01 +0000 (0:00:00.563) 0:02:37.491 ********** 2026-03-20 15:52:02.214678 | controller | Friday 20 March 2026 15:52:01 +0000 (0:00:00.563) 0:02:37.489 ********** 2026-03-20 15:52:02.214715 | controller | ok: [controller] 2026-03-20 15:52:02.798013 | controller | 2026-03-20 15:52:02.798042 | controller | TASK [openshift_login : Append the KUBECONFIG to the install yamls parameters content={{ 2026-03-20 15:52:02.798049 | controller | cifmw_openshift_login_install_yamls_artifacts_slurp['content'] | b64decode | from_yaml | 2026-03-20 15:52:02.798053 | controller | combine( 2026-03-20 15:52:02.798058 | controller | { 2026-03-20 15:52:02.798062 | controller | 'cifmw_install_yamls_environment': { 2026-03-20 15:52:02.798067 | controller | 'KUBECONFIG': cifmw_openshift_login_kubeconfig 2026-03-20 15:52:02.798072 | controller | } 2026-03-20 15:52:02.798076 | controller | }, recursive=true) | to_nice_yaml 2026-03-20 15:52:02.798081 | controller | }}, dest={{ cifmw_basedir }}/artifacts/parameters/install-yamls-params.yml, mode=0600] *** 2026-03-20 15:52:02.798085 | controller | Friday 20 March 2026 15:52:02 +0000 (0:00:00.347) 0:02:37.838 ********** 2026-03-20 15:52:02.798090 | controller | Friday 20 March 2026 15:52:02 +0000 (0:00:00.347) 0:02:37.837 ********** 2026-03-20 15:52:02.798099 | controller | changed: [controller] 2026-03-20 15:52:03.005238 | controller | 2026-03-20 15:52:03.005276 | controller | TASK [openshift_setup : Ensure output directory exists path={{ cifmw_openshift_setup_basedir }}/artifacts, state=directory, mode=0755] *** 2026-03-20 15:52:03.005286 | controller | Friday 20 March 2026 15:52:02 +0000 (0:00:00.583) 0:02:38.422 ********** 2026-03-20 15:52:03.005292 | controller | Friday 20 March 2026 15:52:02 +0000 (0:00:00.583) 0:02:38.421 ********** 2026-03-20 15:52:03.005304 | controller | ok: [controller] 2026-03-20 15:52:03.036831 | controller | 2026-03-20 15:52:03.036862 | controller | TASK [openshift_setup : Fetch namespaces to create cifmw_openshift_setup_namespaces={{ 2026-03-20 15:52:03.036868 | controller | (( 2026-03-20 15:52:03.036873 | controller | ([cifmw_install_yamls_defaults['NAMESPACE']] + 2026-03-20 15:52:03.036878 | controller | ([cifmw_install_yamls_defaults['OPERATOR_NAMESPACE']] if 'OPERATOR_NAMESPACE' is in cifmw_install_yamls_defaults else []) 2026-03-20 15:52:03.036883 | controller | ) if cifmw_install_yamls_defaults is defined else [] 2026-03-20 15:52:03.036887 | controller | ) + cifmw_openshift_setup_create_namespaces) | unique 2026-03-20 15:52:03.036892 | controller | }}] *** 2026-03-20 15:52:03.036896 | controller | Friday 20 March 2026 15:52:03 +0000 (0:00:00.207) 0:02:38.629 ********** 2026-03-20 15:52:03.036901 | controller | Friday 20 March 2026 15:52:03 +0000 (0:00:00.207) 0:02:38.628 ********** 2026-03-20 15:52:03.036909 | controller | ok: [controller] 2026-03-20 15:52:04.904395 | controller | 2026-03-20 15:52:04.904431 | controller | TASK [openshift_setup : Create required namespaces kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit) }}, name={{ item }}, kind=Namespace, state=present] *** 2026-03-20 15:52:04.904443 | controller | Friday 20 March 2026 15:52:03 +0000 (0:00:00.031) 0:02:38.661 ********** 2026-03-20 15:52:04.904450 | controller | Friday 20 March 2026 15:52:03 +0000 (0:00:00.031) 0:02:38.659 ********** 2026-03-20 15:52:04.904461 | controller | changed: [controller] => (item=openstack) 2026-03-20 15:52:04.926736 | controller | changed: [controller] => (item=openstack-operators) 2026-03-20 15:52:04.926774 | controller | 2026-03-20 15:52:04.926782 | controller | TASK [openshift_setup : Get internal OpenShift registry route kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, kind=Route, name=default-route, namespace=openshift-image-registry] *** 2026-03-20 15:52:04.926789 | controller | Friday 20 March 2026 15:52:04 +0000 (0:00:01.867) 0:02:40.528 ********** 2026-03-20 15:52:04.926796 | controller | Friday 20 March 2026 15:52:04 +0000 (0:00:01.867) 0:02:40.527 ********** 2026-03-20 15:52:04.926807 | controller | skipping: [controller] 2026-03-20 15:52:04.956952 | controller | 2026-03-20 15:52:04.956992 | controller | TASK [openshift_setup : Allow anonymous image-pulls in CRC registry for targeted namespaces state=present, kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, definition={'kind': 'RoleBinding', 'apiVersion': 'rbac.authorization.k8s.io/v1', 'metadata': {'name': 'system:image-puller', 'namespace': '{{ item }}'}, 'subjects': [{'kind': 'User', 'name': 'system:anonymous'}, {'kind': 'User', 'name': 'system:unauthenticated'}], 'roleRef': {'kind': 'ClusterRole', 'name': 'system:image-puller'}}] *** 2026-03-20 15:52:04.957002 | controller | Friday 20 March 2026 15:52:04 +0000 (0:00:00.022) 0:02:40.550 ********** 2026-03-20 15:52:04.957009 | controller | Friday 20 March 2026 15:52:04 +0000 (0:00:00.022) 0:02:40.549 ********** 2026-03-20 15:52:04.957021 | controller | skipping: [controller] => (item=openstack) 2026-03-20 15:52:04.981776 | controller | skipping: [controller] => (item=openstack-operators) 2026-03-20 15:52:04.981808 | controller | skipping: [controller] 2026-03-20 15:52:04.981815 | controller | 2026-03-20 15:52:04.981820 | controller | TASK [openshift_setup : Wait for the image registry to be ready kind=Deployment, name=image-registry, namespace=openshift-image-registry, kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, wait=True, wait_sleep=10, wait_timeout=600, wait_condition={'type': 'Available', 'status': 'True'}] *** 2026-03-20 15:52:04.981825 | controller | Friday 20 March 2026 15:52:04 +0000 (0:00:00.030) 0:02:40.581 ********** 2026-03-20 15:52:04.981830 | controller | Friday 20 March 2026 15:52:04 +0000 (0:00:00.030) 0:02:40.579 ********** 2026-03-20 15:52:04.981839 | controller | skipping: [controller] 2026-03-20 15:52:05.004417 | controller | 2026-03-20 15:52:05.004451 | controller | TASK [openshift_setup : Login into OpenShift internal registry output_dir={{ cifmw_openshift_setup_basedir }}/artifacts, script=podman login -u {{ cifmw_openshift_user }} -p {{ cifmw_openshift_token }} {%- if cifmw_openshift_setup_skip_internal_registry_tls_verify|bool %} --tls-verify=false {%- endif %} {{ cifmw_openshift_setup_registry_default_route.resources[0].spec.host }}] *** 2026-03-20 15:52:05.004457 | controller | Friday 20 March 2026 15:52:04 +0000 (0:00:00.024) 0:02:40.606 ********** 2026-03-20 15:52:05.004463 | controller | Friday 20 March 2026 15:52:04 +0000 (0:00:00.024) 0:02:40.604 ********** 2026-03-20 15:52:05.004474 | controller | skipping: [controller] 2026-03-20 15:52:05.025145 | controller | 2026-03-20 15:52:05.025174 | controller | TASK [Ensure we have custom CA installed on host role=install_ca] ************** 2026-03-20 15:52:05.025180 | controller | Friday 20 March 2026 15:52:05 +0000 (0:00:00.022) 0:02:40.628 ********** 2026-03-20 15:52:05.025185 | controller | Friday 20 March 2026 15:52:05 +0000 (0:00:00.022) 0:02:40.627 ********** 2026-03-20 15:52:05.025194 | controller | skipping: [controller] 2026-03-20 15:52:05.046485 | controller | 2026-03-20 15:52:05.046515 | controller | TASK [openshift_setup : Update ca bundle _raw_params=update-ca-trust extract] *** 2026-03-20 15:52:05.046525 | controller | Friday 20 March 2026 15:52:05 +0000 (0:00:00.020) 0:02:40.649 ********** 2026-03-20 15:52:05.046530 | controller | Friday 20 March 2026 15:52:05 +0000 (0:00:00.020) 0:02:40.648 ********** 2026-03-20 15:52:05.046538 | controller | skipping: [controller] 2026-03-20 15:52:05.067428 | controller | 2026-03-20 15:52:05.067456 | controller | TASK [openshift_setup : Slurp CAs file src={{ cifmw_openshift_setup_ca_bundle_path }}] *** 2026-03-20 15:52:05.067464 | controller | Friday 20 March 2026 15:52:05 +0000 (0:00:00.021) 0:02:40.670 ********** 2026-03-20 15:52:05.067469 | controller | Friday 20 March 2026 15:52:05 +0000 (0:00:00.021) 0:02:40.669 ********** 2026-03-20 15:52:05.067477 | controller | skipping: [controller] 2026-03-20 15:52:05.091008 | controller | 2026-03-20 15:52:05.091039 | controller | TASK [openshift_setup : Create config map with registry CAs kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, definition={'apiVersion': 'v1', 'kind': 'ConfigMap', 'metadata': {'namespace': 'openshift-config', 'name': 'registry-cas'}, 'data': '{{ _config_map_data | items2dict }}'}] *** 2026-03-20 15:52:05.091046 | controller | Friday 20 March 2026 15:52:05 +0000 (0:00:00.020) 0:02:40.691 ********** 2026-03-20 15:52:05.091051 | controller | Friday 20 March 2026 15:52:05 +0000 (0:00:00.020) 0:02:40.690 ********** 2026-03-20 15:52:05.091059 | controller | skipping: [controller] 2026-03-20 15:52:05.113749 | controller | 2026-03-20 15:52:05.113781 | controller | TASK [openshift_setup : Install Red Hat CA for pulling images from internal registry kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, merge_type=merge, definition={'apiVersion': 'config.openshift.io/v1', 'kind': 'Image', 'metadata': {'name': 'cluster'}, 'spec': {'additionalTrustedCA': {'name': 'registry-cas'}}}] *** 2026-03-20 15:52:05.113790 | controller | Friday 20 March 2026 15:52:05 +0000 (0:00:00.023) 0:02:40.715 ********** 2026-03-20 15:52:05.113795 | controller | Friday 20 March 2026 15:52:05 +0000 (0:00:00.023) 0:02:40.714 ********** 2026-03-20 15:52:05.113804 | controller | skipping: [controller] 2026-03-20 15:52:05.136644 | controller | 2026-03-20 15:52:05.136681 | controller | TASK [openshift_setup : Add insecure registry kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, merge_type=merge, definition={'apiVersion': 'config.openshift.io/v1', 'kind': 'Image', 'metadata': {'name': 'cluster'}, 'spec': {'registrySources': {'insecureRegistries': ['{{ cifmw_update_containers_registry }}'], 'allowedRegistries': '{{ all_registries }}'}}}] *** 2026-03-20 15:52:05.136689 | controller | Friday 20 March 2026 15:52:05 +0000 (0:00:00.022) 0:02:40.738 ********** 2026-03-20 15:52:05.136725 | controller | Friday 20 March 2026 15:52:05 +0000 (0:00:00.022) 0:02:40.736 ********** 2026-03-20 15:52:05.136735 | controller | skipping: [controller] 2026-03-20 15:52:05.161841 | controller | 2026-03-20 15:52:05.161878 | controller | TASK [openshift_setup : Create a ICSP with repository digest mirrors kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, definition={'apiVersion': 'operator.openshift.io/v1alpha1', 'kind': 'ImageContentSourcePolicy', 'metadata': {'name': 'registry-digest-mirrors'}, 'spec': {'repositoryDigestMirrors': '{{ cifmw_openshift_setup_digest_mirrors }}'}}] *** 2026-03-20 15:52:05.161885 | controller | Friday 20 March 2026 15:52:05 +0000 (0:00:00.022) 0:02:40.760 ********** 2026-03-20 15:52:05.161891 | controller | Friday 20 March 2026 15:52:05 +0000 (0:00:00.022) 0:02:40.759 ********** 2026-03-20 15:52:05.161920 | controller | skipping: [controller] 2026-03-20 15:52:06.112178 | controller | 2026-03-20 15:52:06.112214 | controller | TASK [openshift_setup : Gather network.operator info kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, api_version=operator.openshift.io/v1, kind=Network, name=cluster] *** 2026-03-20 15:52:06.112226 | controller | Friday 20 March 2026 15:52:05 +0000 (0:00:00.025) 0:02:40.786 ********** 2026-03-20 15:52:06.112233 | controller | Friday 20 March 2026 15:52:05 +0000 (0:00:00.025) 0:02:40.784 ********** 2026-03-20 15:52:06.112245 | controller | ok: [controller] 2026-03-20 15:52:07.048616 | controller | 2026-03-20 15:52:07.048704 | controller | TASK [openshift_setup : Patch network operator api_version=operator.openshift.io/v1, kubeconfig={{ cifmw_openshift_kubeconfig }}, kind=Network, name=cluster, persist_config=True, patch=[{'path': '/spec/defaultNetwork/ovnKubernetesConfig/gatewayConfig/routingViaHost', 'value': True, 'op': 'replace'}, {'path': '/spec/defaultNetwork/ovnKubernetesConfig/gatewayConfig/ipForwarding', 'value': 'Global', 'op': 'replace'}]] *** 2026-03-20 15:52:07.048724 | controller | Friday 20 March 2026 15:52:06 +0000 (0:00:00.950) 0:02:41.736 ********** 2026-03-20 15:52:07.048730 | controller | Friday 20 March 2026 15:52:06 +0000 (0:00:00.950) 0:02:41.735 ********** 2026-03-20 15:52:07.048745 | controller | changed: [controller] 2026-03-20 15:52:07.879551 | controller | 2026-03-20 15:52:07.879594 | controller | TASK [openshift_setup : Patch samples registry configuration kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, api_version=samples.operator.openshift.io/v1, kind=Config, name=cluster, patch=[{'op': 'replace', 'path': '/spec/samplesRegistry', 'value': '{{ cifmw_openshift_setup_samples_registry }}'}]] *** 2026-03-20 15:52:07.879600 | controller | Friday 20 March 2026 15:52:07 +0000 (0:00:00.936) 0:02:42.672 ********** 2026-03-20 15:52:07.879605 | controller | Friday 20 March 2026 15:52:07 +0000 (0:00:00.936) 0:02:42.671 ********** 2026-03-20 15:52:07.879614 | controller | changed: [controller] 2026-03-20 15:52:07.901255 | controller | 2026-03-20 15:52:07.901289 | controller | TASK [openshift_setup : Delete the pods from openshift-marketplace namespace kind=Pod, state=absent, delete_all=True, kubeconfig={{ cifmw_openshift_kubeconfig }}, namespace=openshift-marketplace] *** 2026-03-20 15:52:07.901296 | controller | Friday 20 March 2026 15:52:07 +0000 (0:00:00.831) 0:02:43.503 ********** 2026-03-20 15:52:07.901301 | controller | Friday 20 March 2026 15:52:07 +0000 (0:00:00.831) 0:02:43.502 ********** 2026-03-20 15:52:07.901310 | controller | skipping: [controller] 2026-03-20 15:52:07.929583 | controller | 2026-03-20 15:52:07.929617 | controller | TASK [openshift_setup : Wait for openshift-marketplace pods to be running _raw_params=oc wait pod --all --for=condition=Ready -n openshift-marketplace --timeout=1m] *** 2026-03-20 15:52:07.929624 | controller | Friday 20 March 2026 15:52:07 +0000 (0:00:00.021) 0:02:43.525 ********** 2026-03-20 15:52:07.929629 | controller | Friday 20 March 2026 15:52:07 +0000 (0:00:00.021) 0:02:43.524 ********** 2026-03-20 15:52:07.929638 | controller | skipping: [controller] 2026-03-20 15:52:08.016846 | controller | 2026-03-20 15:52:08.016884 | controller | TASK [Deploy Observability operator. name=openshift_obs] *********************** 2026-03-20 15:52:08.016892 | controller | Friday 20 March 2026 15:52:07 +0000 (0:00:00.028) 0:02:43.553 ********** 2026-03-20 15:52:08.016899 | controller | Friday 20 March 2026 15:52:07 +0000 (0:00:00.028) 0:02:43.552 ********** 2026-03-20 15:52:08.016911 | controller | 2026-03-20 15:52:08.865149 | controller | TASK [openshift_obs : Install cluster observability operator. definition={{cifmw_openshift_obs_definition }}, kubeconfig={{ cifmw_openshift_kubeconfig }}, state=present] *** 2026-03-20 15:52:08.865184 | controller | Friday 20 March 2026 15:52:08 +0000 (0:00:00.086) 0:02:43.640 ********** 2026-03-20 15:52:08.865191 | controller | Friday 20 March 2026 15:52:08 +0000 (0:00:00.086) 0:02:43.639 ********** 2026-03-20 15:52:08.865200 | controller | changed: [controller] 2026-03-20 15:52:59.913725 | controller | 2026-03-20 15:52:59.913766 | controller | TASK [openshift_obs : Wait for observability operator deployment kind=Deployment, namespace=openshift-operators, name=observability-operator, wait=True, wait_timeout=300, wait_condition={'type': 'Available', 'status': 'True'}, kubeconfig={{ cifmw_openshift_kubeconfig }}] *** 2026-03-20 15:52:59.913775 | controller | Friday 20 March 2026 15:52:08 +0000 (0:00:00.848) 0:02:44.489 ********** 2026-03-20 15:52:59.913787 | controller | Friday 20 March 2026 15:52:08 +0000 (0:00:00.848) 0:02:44.488 ********** 2026-03-20 15:52:59.913798 | controller | ok: [controller] 2026-03-20 15:53:00.809095 | controller | 2026-03-20 15:53:00.809137 | controller | TASK [openshift_obs : Wait for observability-operator pod kind=Pod, namespace=openshift-operators, label_selectors=['app.kubernetes.io/name = observability-operator'], wait=True, wait_timeout=300, wait_condition={'type': 'Ready', 'status': 'True'}, kubeconfig={{ cifmw_openshift_kubeconfig }}] *** 2026-03-20 15:53:00.809150 | controller | Friday 20 March 2026 15:52:59 +0000 (0:00:51.048) 0:03:35.537 ********** 2026-03-20 15:53:00.809160 | controller | Friday 20 March 2026 15:52:59 +0000 (0:00:51.048) 0:03:35.536 ********** 2026-03-20 15:53:00.809176 | controller | ok: [controller] 2026-03-20 15:53:00.837517 | controller | 2026-03-20 15:53:00.837548 | controller | TASK [Deploy Metal3 BMHs name=deploy_bmh] ************************************** 2026-03-20 15:53:00.837554 | controller | Friday 20 March 2026 15:53:00 +0000 (0:00:00.895) 0:03:36.433 ********** 2026-03-20 15:53:00.837561 | controller | Friday 20 March 2026 15:53:00 +0000 (0:00:00.895) 0:03:36.432 ********** 2026-03-20 15:53:00.837570 | controller | skipping: [controller] 2026-03-20 15:53:00.865928 | controller | 2026-03-20 15:53:00.865963 | controller | TASK [Install certmanager operator role name=cert_manager] ********************* 2026-03-20 15:53:00.865970 | controller | Friday 20 March 2026 15:53:00 +0000 (0:00:00.028) 0:03:36.461 ********** 2026-03-20 15:53:00.865975 | controller | Friday 20 March 2026 15:53:00 +0000 (0:00:00.028) 0:03:36.460 ********** 2026-03-20 15:53:00.865986 | controller | skipping: [controller] 2026-03-20 15:53:00.894175 | controller | 2026-03-20 15:53:00.894213 | controller | TASK [Configure hosts networking using nmstate name=ci_nmstate] **************** 2026-03-20 15:53:00.894222 | controller | Friday 20 March 2026 15:53:00 +0000 (0:00:00.028) 0:03:36.490 ********** 2026-03-20 15:53:00.894228 | controller | Friday 20 March 2026 15:53:00 +0000 (0:00:00.028) 0:03:36.489 ********** 2026-03-20 15:53:00.894239 | controller | skipping: [controller] 2026-03-20 15:53:00.921651 | controller | 2026-03-20 15:53:00.921689 | controller | TASK [Configure multus networks name=ci_multus] ******************************** 2026-03-20 15:53:00.921730 | controller | Friday 20 March 2026 15:53:00 +0000 (0:00:00.028) 0:03:36.518 ********** 2026-03-20 15:53:00.921740 | controller | Friday 20 March 2026 15:53:00 +0000 (0:00:00.028) 0:03:36.517 ********** 2026-03-20 15:53:00.921757 | controller | skipping: [controller] 2026-03-20 15:53:00.948374 | controller | 2026-03-20 15:53:00.948412 | controller | TASK [Deploy Sushy Emulator service pod name=sushy_emulator] ******************* 2026-03-20 15:53:00.948422 | controller | Friday 20 March 2026 15:53:00 +0000 (0:00:00.027) 0:03:36.545 ********** 2026-03-20 15:53:00.948429 | controller | Friday 20 March 2026 15:53:00 +0000 (0:00:00.027) 0:03:36.544 ********** 2026-03-20 15:53:00.948440 | controller | skipping: [controller] 2026-03-20 15:53:00.977661 | controller | 2026-03-20 15:53:00.977731 | controller | TASK [Setup Libvirt on controller name=libvirt_manager] ************************ 2026-03-20 15:53:00.977743 | controller | Friday 20 March 2026 15:53:00 +0000 (0:00:00.026) 0:03:36.572 ********** 2026-03-20 15:53:00.977750 | controller | Friday 20 March 2026 15:53:00 +0000 (0:00:00.026) 0:03:36.571 ********** 2026-03-20 15:53:00.977765 | controller | skipping: [controller] 2026-03-20 15:53:01.004377 | controller | 2026-03-20 15:53:01.004417 | controller | TASK [Prepare container package builder name=pkg_build] ************************ 2026-03-20 15:53:01.004427 | controller | Friday 20 March 2026 15:53:00 +0000 (0:00:00.029) 0:03:36.601 ********** 2026-03-20 15:53:01.004434 | controller | Friday 20 March 2026 15:53:00 +0000 (0:00:00.029) 0:03:36.600 ********** 2026-03-20 15:53:01.004447 | controller | skipping: [controller] 2026-03-20 15:53:01.061387 | controller | 2026-03-20 15:53:01.061425 | controller | TASK [run_hook : Assert parameters are valid quiet=True, that=['_list_hooks is not string', '_list_hooks is not mapping', '_list_hooks is iterable', '(hooks | default([])) is not string', '(hooks | default([])) is not mapping', '(hooks | default([])) is iterable']] *** 2026-03-20 15:53:01.061435 | controller | Friday 20 March 2026 15:53:01 +0000 (0:00:00.026) 0:03:36.628 ********** 2026-03-20 15:53:01.061447 | controller | Friday 20 March 2026 15:53:01 +0000 (0:00:00.026) 0:03:36.627 ********** 2026-03-20 15:53:01.061459 | controller | ok: [controller] 2026-03-20 15:53:01.157472 | controller | 2026-03-20 15:53:01.157504 | controller | TASK [run_hook : Assert single hooks are all mappings quiet=True, that=['_not_mapping_hooks | length == 0'], msg=All single hooks must be a list of mappings or a mapping.] *** 2026-03-20 15:53:01.157511 | controller | Friday 20 March 2026 15:53:01 +0000 (0:00:00.057) 0:03:36.685 ********** 2026-03-20 15:53:01.157516 | controller | Friday 20 March 2026 15:53:01 +0000 (0:00:00.057) 0:03:36.684 ********** 2026-03-20 15:53:01.157525 | controller | ok: [controller] 2026-03-20 15:53:01.261731 | controller | 2026-03-20 15:53:01.261768 | controller | TASK [run_hook : Loop on hooks for post_infra _raw_params={{ hook.type }}.yml] *** 2026-03-20 15:53:01.261776 | controller | Friday 20 March 2026 15:53:01 +0000 (0:00:00.096) 0:03:36.781 ********** 2026-03-20 15:53:01.261783 | controller | Friday 20 March 2026 15:53:01 +0000 (0:00:00.096) 0:03:36.780 ********** 2026-03-20 15:53:01.261793 | controller | skipping: [controller] 2026-03-20 15:53:01.314614 | controller | 2026-03-20 15:53:01.314647 | controller | TASK [Load parameters dir={{ item }}] ****************************************** 2026-03-20 15:53:01.314653 | controller | Friday 20 March 2026 15:53:01 +0000 (0:00:00.104) 0:03:36.886 ********** 2026-03-20 15:53:01.314658 | controller | Friday 20 March 2026 15:53:01 +0000 (0:00:00.104) 0:03:36.884 ********** 2026-03-20 15:53:01.314669 | controller | ok: [controller] => (item=/home/zuul/ci-framework-data/artifacts/parameters) 2026-03-20 15:53:01.390614 | controller | ok: [controller] => (item=/etc/ci/env) 2026-03-20 15:53:01.390645 | controller | 2026-03-20 15:53:01.390653 | controller | TASK [Ensure that the isolated net was configured for crc that=['crc_ci_bootstrap_networks_out is defined', 'crc_ci_bootstrap_networks_out[_crc_hostname] is defined', "crc_ci_bootstrap_networks_out[_crc_hostname]['default'] is defined"]] *** 2026-03-20 15:53:01.390664 | controller | Friday 20 March 2026 15:53:01 +0000 (0:00:00.052) 0:03:36.938 ********** 2026-03-20 15:53:01.390671 | controller | Friday 20 March 2026 15:53:01 +0000 (0:00:00.052) 0:03:36.937 ********** 2026-03-20 15:53:01.390682 | controller | ok: [controller] => 2026-03-20 15:53:01.450450 | controller | changed: false 2026-03-20 15:53:01.450482 | controller | msg: All assertions passed 2026-03-20 15:53:01.450488 | controller | 2026-03-20 15:53:01.450493 | controller | TASK [Set facts for further usage within the framework cifmw_edpm_prepare_extra_vars={'NNCP_INTERFACE': '{{ crc_ci_bootstrap_networks_out.crc.default.iface }}', 'NETWORK_MTU': '{{ crc_ci_bootstrap_networks_out.crc.default.mtu }}', 'NNCP_DNS_SERVER': "{{\n cifmw_nncp_dns_server |\n default(crc_ci_bootstrap_networks_out[_crc_hostname].default.ip) |\n split('/') | first\n}}"}] *** 2026-03-20 15:53:01.450499 | controller | Friday 20 March 2026 15:53:01 +0000 (0:00:00.076) 0:03:37.015 ********** 2026-03-20 15:53:01.450504 | controller | Friday 20 March 2026 15:53:01 +0000 (0:00:00.076) 0:03:37.013 ********** 2026-03-20 15:53:01.450513 | controller | ok: [controller] 2026-03-20 15:53:01.495143 | controller | 2026-03-20 15:53:01.495179 | controller | PLAY [Deploy Openstack Operators] ********************************************** 2026-03-20 15:53:01.495187 | controller | 2026-03-20 15:53:01.495194 | controller | TASK [Use the locally built operators if any _local_operators_indexes={{ 2026-03-20 15:53:01.495201 | controller | _local_operators_indexes|default({}) | 2026-03-20 15:53:01.495207 | controller | combine({ item.key.split('-')[0]|upper+'_IMG': 2026-03-20 15:53:01.495213 | controller | cifmw_operator_build_output['operators'][item.key].image_catalog}) 2026-03-20 15:53:01.495219 | controller | }}] *** 2026-03-20 15:53:01.495225 | controller | Friday 20 March 2026 15:53:01 +0000 (0:00:00.059) 0:03:37.074 ********** 2026-03-20 15:53:01.495232 | controller | Friday 20 March 2026 15:53:01 +0000 (0:00:00.059) 0:03:37.073 ********** 2026-03-20 15:53:01.495245 | controller | ok: [controller] => (item={'key': 'openstack-operator', 'value': {'git_commit_hash': '36856d22fbbd028e148ba6b5277b8d8be928cf7c', 'git_src_dir': '/home/zuul-worker/src/github.com/openstack-k8s-operators/openstack-operator', 'image': '38.102.83.147:5001/openstack-k8s-operators/openstack-operator:36856d22fbbd028e148ba6b5277b8d8be928cf7c', 'image_bundle': '38.102.83.147:5001/openstack-k8s-operators/openstack-operator-bundle:36856d22fbbd028e148ba6b5277b8d8be928cf7c', 'image_catalog': '38.102.83.147:5001/openstack-k8s-operators/openstack-operator-index:36856d22fbbd028e148ba6b5277b8d8be928cf7c'}}) 2026-03-20 15:53:01.521237 | controller | ok: [controller] => (item={'key': 'watcher-operator', 'value': {'git_commit_hash': 'ee00c2d330b27d46c48ac29a20680b56ca50df3c', 'git_src_dir': '/home/zuul-worker/src/github.com/openstack-k8s-operators/watcher-operator', 'image': '38.102.83.147:5001/openstack-k8s-operators/watcher-operator:ee00c2d330b27d46c48ac29a20680b56ca50df3c', 'image_bundle': '38.102.83.147:5001/openstack-k8s-operators/watcher-operator-bundle:ee00c2d330b27d46c48ac29a20680b56ca50df3c', 'image_catalog': '38.102.83.147:5001/openstack-k8s-operators/watcher-operator-index:ee00c2d330b27d46c48ac29a20680b56ca50df3c'}}) 2026-03-20 15:53:01.521272 | controller | 2026-03-20 15:53:01.521278 | controller | TASK [Set install_yamls Makefile environment variables cifmw_edpm_prepare_common_env={{ 2026-03-20 15:53:01.521283 | controller | cifmw_install_yamls_environment | 2026-03-20 15:53:01.521289 | controller | combine({'PATH': cifmw_path}) | 2026-03-20 15:53:01.521294 | controller | combine(cifmw_edpm_prepare_extra_vars | default({})) 2026-03-20 15:53:01.521298 | controller | }}, cifmw_edpm_prepare_operators_build_output={{ operators_build_output }}, cifmw_edpm_prepare_make_openstack_env={{ _local_operators_indexes | combine(_openstack_operator_images) }}] *** 2026-03-20 15:53:01.521303 | controller | Friday 20 March 2026 15:53:01 +0000 (0:00:00.044) 0:03:37.119 ********** 2026-03-20 15:53:01.521308 | controller | Friday 20 March 2026 15:53:01 +0000 (0:00:00.044) 0:03:37.118 ********** 2026-03-20 15:53:01.521316 | controller | ok: [controller] 2026-03-20 15:53:01.845029 | controller | 2026-03-20 15:53:01.845058 | controller | TASK [detect if openstack operator is installed _raw_params=oc get sub --ignore-not-found=true -n openstack-operators -o name openstack-operator] *** 2026-03-20 15:53:01.845066 | controller | Friday 20 March 2026 15:53:01 +0000 (0:00:00.026) 0:03:37.145 ********** 2026-03-20 15:53:01.845071 | controller | Friday 20 March 2026 15:53:01 +0000 (0:00:00.026) 0:03:37.144 ********** 2026-03-20 15:53:01.845080 | controller | changed: [controller] 2026-03-20 15:53:01.875883 | controller | 2026-03-20 15:53:01.875922 | controller | TASK [Install openstack operator and wait for the csv to succeed name=install_yamls_makes, tasks_from=make_openstack_init] *** 2026-03-20 15:53:01.875930 | controller | Friday 20 March 2026 15:53:01 +0000 (0:00:00.323) 0:03:37.469 ********** 2026-03-20 15:53:01.875938 | controller | Friday 20 March 2026 15:53:01 +0000 (0:00:00.323) 0:03:37.468 ********** 2026-03-20 15:53:01.875950 | controller | 2026-03-20 15:53:01.908182 | controller | TASK [install_yamls_makes : Debug make_openstack_init_env var=make_openstack_init_env] *** 2026-03-20 15:53:01.908235 | controller | Friday 20 March 2026 15:53:01 +0000 (0:00:00.030) 0:03:37.500 ********** 2026-03-20 15:53:01.908242 | controller | Friday 20 March 2026 15:53:01 +0000 (0:00:00.030) 0:03:37.498 ********** 2026-03-20 15:53:01.908251 | controller | ok: [controller] => 2026-03-20 15:53:01.930411 | controller | make_openstack_init_env: 2026-03-20 15:53:01.930445 | controller | BMO_SETUP: false 2026-03-20 15:53:01.930451 | controller | CHECKOUT_FROM_OPENSTACK_REF: 'true' 2026-03-20 15:53:01.930456 | controller | KUBECONFIG: /home/zuul/.crc/machines/crc/kubeconfig 2026-03-20 15:53:01.930461 | controller | NETWORK_MTU: 1500 2026-03-20 15:53:01.930465 | controller | NNCP_DNS_SERVER: 192.168.122.10 2026-03-20 15:53:01.930472 | controller | NNCP_INTERFACE: ens7 2026-03-20 15:53:01.930477 | controller | OPENSTACK_BUNDLE_IMG: 38.102.83.147:5001/openstack-k8s-operators/openstack-operator-bundle:36856d22fbbd028e148ba6b5277b8d8be928cf7c 2026-03-20 15:53:01.930483 | controller | OPENSTACK_IMG: 38.102.83.147:5001/openstack-k8s-operators/openstack-operator-index:36856d22fbbd028e148ba6b5277b8d8be928cf7c 2026-03-20 15:53:01.930488 | controller | OPENSTACK_K8S_BRANCH: main 2026-03-20 15:53:01.930492 | controller | OUT: /home/zuul/ci-framework-data/artifacts/manifests 2026-03-20 15:53:01.930497 | controller | OUTPUT_DIR: /home/zuul/ci-framework-data/artifacts/edpm 2026-03-20 15:53:01.930505 | controller | PATH: /home/zuul/.crc/bin:/home/zuul/.crc/bin/oc:/home/zuul/bin:/home/zuul/.local/bin:/home/zuul/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin 2026-03-20 15:53:01.930540 | controller | WATCHER_BRANCH: '' 2026-03-20 15:53:01.930547 | controller | WATCHER_IMG: 38.102.83.147:5001/openstack-k8s-operators/watcher-operator-index:ee00c2d330b27d46c48ac29a20680b56ca50df3c 2026-03-20 15:53:01.930552 | controller | WATCHER_REPO: /home/zuul/src/github.com/openstack-k8s-operators/watcher-operator 2026-03-20 15:53:01.930557 | controller | 2026-03-20 15:53:01.930561 | controller | TASK [install_yamls_makes : Debug make_openstack_init_params var=make_openstack_init_params] *** 2026-03-20 15:53:01.930566 | controller | Friday 20 March 2026 15:53:01 +0000 (0:00:00.032) 0:03:37.532 ********** 2026-03-20 15:53:01.930570 | controller | Friday 20 March 2026 15:53:01 +0000 (0:00:00.032) 0:03:37.531 ********** 2026-03-20 15:53:01.930579 | controller | skipping: [controller] 2026-03-20 15:57:52.745862 | controller | 2026-03-20 15:57:52.745977 | controller | TASK [install_yamls_makes : Run openstack_init output_dir={{ cifmw_basedir }}/artifacts, chdir=/home/zuul/src/github.com/openstack-k8s-operators/install_yamls, script=make openstack_init, dry_run={{ make_openstack_init_dryrun|default(false)|bool }}, extra_args={{ dict((make_openstack_init_env|default({})), **(make_openstack_init_params|default({}))) }}] *** 2026-03-20 15:57:52.745985 | controller | Friday 20 March 2026 15:53:01 +0000 (0:00:00.022) 0:03:37.554 ********** 2026-03-20 15:57:52.745990 | controller | Friday 20 March 2026 15:53:01 +0000 (0:00:00.022) 0:03:37.553 ********** 2026-03-20 15:57:52.746009 | controller | [WARNING]: conditional statements should not include jinja2 templating 2026-03-20 15:57:52.765941 | controller | delimiters such as {{ }} or {% %}. Found: {{ make_openstack_init_until | 2026-03-20 15:57:52.765977 | controller | default(true) }} 2026-03-20 15:57:52.765989 | controller | Follow script's output here: /home/zuul/ci-framework-data/logs/ci_script_003_run_openstack.log 2026-03-20 15:57:52.806988 | controller | changed: [controller] 2026-03-20 15:57:52.807038 | controller | 2026-03-20 15:57:52.807045 | controller | TASK [Run hooks after installing openstack name=run_hook] ********************** 2026-03-20 15:57:52.807051 | controller | Friday 20 March 2026 15:57:52 +0000 (0:04:50.835) 0:08:28.390 ********** 2026-03-20 15:57:52.807056 | controller | Friday 20 March 2026 15:57:52 +0000 (0:04:50.835) 0:08:28.389 ********** 2026-03-20 15:57:52.807067 | controller | 2026-03-20 15:57:52.893686 | controller | TASK [run_hook : Assert parameters are valid quiet=True, that=['_list_hooks is not string', '_list_hooks is not mapping', '_list_hooks is iterable', '(hooks | default([])) is not string', '(hooks | default([])) is not mapping', '(hooks | default([])) is iterable']] *** 2026-03-20 15:57:52.893768 | controller | Friday 20 March 2026 15:57:52 +0000 (0:00:00.040) 0:08:28.430 ********** 2026-03-20 15:57:52.893777 | controller | Friday 20 March 2026 15:57:52 +0000 (0:00:00.040) 0:08:28.429 ********** 2026-03-20 15:57:52.893791 | controller | ok: [controller] 2026-03-20 15:57:53.011374 | controller | 2026-03-20 15:57:53.011424 | controller | TASK [run_hook : Assert single hooks are all mappings quiet=True, that=['_not_mapping_hooks | length == 0'], msg=All single hooks must be a list of mappings or a mapping.] *** 2026-03-20 15:57:53.011432 | controller | Friday 20 March 2026 15:57:52 +0000 (0:00:00.086) 0:08:28.517 ********** 2026-03-20 15:57:53.011438 | controller | Friday 20 March 2026 15:57:52 +0000 (0:00:00.086) 0:08:28.516 ********** 2026-03-20 15:57:53.011449 | controller | ok: [controller] 2026-03-20 15:57:53.161420 | controller | 2026-03-20 15:57:53.161473 | controller | TASK [run_hook : Loop on hooks for post_install_operators_kuttl_from_operator _raw_params={{ hook.type }}.yml] *** 2026-03-20 15:57:53.161482 | controller | Friday 20 March 2026 15:57:53 +0000 (0:00:00.117) 0:08:28.635 ********** 2026-03-20 15:57:53.161489 | controller | Friday 20 March 2026 15:57:53 +0000 (0:00:00.117) 0:08:28.634 ********** 2026-03-20 15:57:53.161503 | controller | included: /home/zuul/src/github.com/openstack-k8s-operators/ci-framework/roles/run_hook/tasks/playbook.yml for controller => (item={'name': 'Deploy watcher operator', 'type': 'playbook', 'source': '/home/zuul/src/github.com/openstack-k8s-operators/watcher-operator/ci/playbooks/deploy_watcher_operator.yaml', 'extra_vars': {'content_provider_os_registry_url': '38.102.83.147:5001/podified-master-centos10', 'watcher_catalog_image': '38.102.83.147:5001/openstack-k8s-operators/watcher-operator-index:ee00c2d330b27d46c48ac29a20680b56ca50df3c', 'watcher_services_tag': 'watcher_latest', 'watcher_repo': '/home/zuul/src/github.com/openstack-k8s-operators/watcher-operator'}}) 2026-03-20 15:57:53.220667 | controller | 2026-03-20 15:57:53.220754 | controller | TASK [run_hook : Set playbook path for Deploy watcher operator cifmw_basedir={{ _bdir }}, hook_name={{ _hook_name }}, playbook_path={{ _play | realpath }}, log_path={{ _bdir }}/logs/{{ step }}_{{ _hook_name }}.log, extra_vars=-e namespace={{ cifmw_openstack_namespace }} {%- if hook.extra_vars is defined and hook.extra_vars|length > 0 -%} {% for key,value in hook.extra_vars.items() -%} {%- if key == 'file' %} -e "@{{ value }}" {%- else %} -e "{{ key }}={{ value }}" {%- endif %} {%- endfor %} {%- endif %}] *** 2026-03-20 15:57:53.220765 | controller | Friday 20 March 2026 15:57:53 +0000 (0:00:00.149) 0:08:28.785 ********** 2026-03-20 15:57:53.220777 | controller | Friday 20 March 2026 15:57:53 +0000 (0:00:00.149) 0:08:28.784 ********** 2026-03-20 15:57:53.220790 | controller | ok: [controller] 2026-03-20 15:57:53.495264 | controller | 2026-03-20 15:57:53.495321 | controller | TASK [run_hook : Get file stat path={{ playbook_path }}] *********************** 2026-03-20 15:57:53.495329 | controller | Friday 20 March 2026 15:57:53 +0000 (0:00:00.059) 0:08:28.844 ********** 2026-03-20 15:57:53.495336 | controller | Friday 20 March 2026 15:57:53 +0000 (0:00:00.059) 0:08:28.843 ********** 2026-03-20 15:57:53.495350 | controller | ok: [controller] 2026-03-20 15:57:53.534208 | controller | 2026-03-20 15:57:53.534271 | controller | TASK [run_hook : Fail if playbook doesn't exist msg=Playbook {{ playbook_path }} doesn't seem to exist.] *** 2026-03-20 15:57:53.534285 | controller | Friday 20 March 2026 15:57:53 +0000 (0:00:00.274) 0:08:29.119 ********** 2026-03-20 15:57:53.534296 | controller | Friday 20 March 2026 15:57:53 +0000 (0:00:00.274) 0:08:29.117 ********** 2026-03-20 15:57:53.534316 | controller | skipping: [controller] 2026-03-20 15:57:53.767766 | controller | 2026-03-20 15:57:53.767811 | controller | TASK [run_hook : Get parameters files paths={{ 2026-03-20 15:57:53.767820 | controller | (cifmw_basedir, 'artifacts/parameters') | path_join 2026-03-20 15:57:53.767826 | controller | }}, file_type=file, patterns=*.yml] *** 2026-03-20 15:57:53.767832 | controller | Friday 20 March 2026 15:57:53 +0000 (0:00:00.038) 0:08:29.157 ********** 2026-03-20 15:57:53.767838 | controller | Friday 20 March 2026 15:57:53 +0000 (0:00:00.038) 0:08:29.156 ********** 2026-03-20 15:57:53.767850 | controller | ok: [controller] 2026-03-20 15:57:53.801936 | controller | 2026-03-20 15:57:53.801970 | controller | TASK [run_hook : Add parameters artifacts as extra variables extra_vars={{ extra_vars }} {% for file in cifmw_run_hook_parameters_files.files %} -e "@{{ file.path }}" {%- endfor %}] *** 2026-03-20 15:57:53.801977 | controller | Friday 20 March 2026 15:57:53 +0000 (0:00:00.234) 0:08:29.392 ********** 2026-03-20 15:57:53.801983 | controller | Friday 20 March 2026 15:57:53 +0000 (0:00:00.234) 0:08:29.390 ********** 2026-03-20 15:57:53.801992 | controller | ok: [controller] 2026-03-20 15:57:54.022167 | controller | 2026-03-20 15:57:54.022208 | controller | TASK [run_hook : Ensure log directory exists path={{ log_path | dirname }}, state=directory, mode=0755] *** 2026-03-20 15:57:54.022218 | controller | Friday 20 March 2026 15:57:53 +0000 (0:00:00.034) 0:08:29.426 ********** 2026-03-20 15:57:54.022225 | controller | Friday 20 March 2026 15:57:53 +0000 (0:00:00.034) 0:08:29.425 ********** 2026-03-20 15:57:54.022238 | controller | ok: [controller] 2026-03-20 15:57:54.263156 | controller | 2026-03-20 15:57:54.263192 | controller | TASK [run_hook : Ensure artifacts directory exists path={{ cifmw_basedir }}/artifacts, state=directory, mode=0755] *** 2026-03-20 15:57:54.263198 | controller | Friday 20 March 2026 15:57:54 +0000 (0:00:00.220) 0:08:29.646 ********** 2026-03-20 15:57:54.263204 | controller | Friday 20 March 2026 15:57:54 +0000 (0:00:00.220) 0:08:29.645 ********** 2026-03-20 15:57:54.263213 | controller | ok: [controller] 2026-03-20 15:58:37.053943 | controller | 2026-03-20 15:58:37.053989 | controller | TASK [run_hook : Run hook without retry - Deploy watcher operator] ************* 2026-03-20 15:58:37.053999 | controller | Friday 20 March 2026 15:57:54 +0000 (0:00:00.241) 0:08:29.887 ********** 2026-03-20 15:58:37.054005 | controller | Friday 20 March 2026 15:57:54 +0000 (0:00:00.241) 0:08:29.886 ********** 2026-03-20 15:58:37.054017 | controller | Follow script's output here: /home/zuul/ci-framework-data/logs/ci_script_004_run_hook_without_retry_deploy.log 2026-03-20 15:58:37.076618 | controller | changed: [controller] 2026-03-20 15:58:37.076656 | controller | 2026-03-20 15:58:37.076664 | controller | TASK [run_hook : Run hook with retry - Deploy watcher operator] **************** 2026-03-20 15:58:37.076669 | controller | Friday 20 March 2026 15:58:37 +0000 (0:00:42.790) 0:09:12.677 ********** 2026-03-20 15:58:37.076673 | controller | Friday 20 March 2026 15:58:37 +0000 (0:00:42.790) 0:09:12.676 ********** 2026-03-20 15:58:37.076683 | controller | skipping: [controller] 2026-03-20 15:58:37.284643 | controller | 2026-03-20 15:58:37.284690 | controller | TASK [run_hook : Check if we have a file path={{ cifmw_basedir }}/artifacts/{{ step }}_{{ hook_name }}.yml] *** 2026-03-20 15:58:37.284719 | controller | Friday 20 March 2026 15:58:37 +0000 (0:00:00.023) 0:09:12.700 ********** 2026-03-20 15:58:37.284732 | controller | Friday 20 March 2026 15:58:37 +0000 (0:00:00.023) 0:09:12.699 ********** 2026-03-20 15:58:37.284750 | controller | ok: [controller] 2026-03-20 15:58:37.308894 | controller | 2026-03-20 15:58:37.308927 | controller | TASK [run_hook : Load generated content in main playbook file={{ cifmw_basedir }}/artifacts/{{ step }}_{{ hook_name }}.yml] *** 2026-03-20 15:58:37.308933 | controller | Friday 20 March 2026 15:58:37 +0000 (0:00:00.206) 0:09:12.907 ********** 2026-03-20 15:58:37.308938 | controller | Friday 20 March 2026 15:58:37 +0000 (0:00:00.206) 0:09:12.906 ********** 2026-03-20 15:58:37.308947 | controller | skipping: [controller] 2026-03-20 16:01:27.342923 | controller | 2026-03-20 16:01:27.342971 | controller | TASK [install kuttl test_suite dependencies chdir={{ ansible_user_dir }}/{{ operator_basedir }}, _raw_params=make kuttl-test-prep] *** 2026-03-20 16:01:27.342980 | controller | Friday 20 March 2026 15:58:37 +0000 (0:00:00.025) 0:09:12.933 ********** 2026-03-20 16:01:27.342987 | controller | Friday 20 March 2026 15:58:37 +0000 (0:00:00.025) 0:09:12.932 ********** 2026-03-20 16:01:27.343000 | controller | changed: [controller] 2026-03-20 16:01:27.443269 | controller | 2026-03-20 16:01:27.443332 | controller | PLAY RECAP ********************************************************************* 2026-03-20 16:01:27.443340 | controller | controller : ok=117 changed=40 unreachable=0 failed=0 skipped=80 rescued=0 ignored=1 2026-03-20 16:01:27.443345 | controller | 2026-03-20 16:01:27.443349 | controller | Friday 20 March 2026 16:01:27 +0000 (0:02:50.032) 0:12:02.965 ********** 2026-03-20 16:01:27.443354 | controller | =============================================================================== 2026-03-20 16:01:27.443358 | controller | install_yamls_makes : Run openstack_init ------------------------------ 290.84s 2026-03-20 16:01:27.443362 | controller | install kuttl test_suite dependencies --------------------------------- 170.03s 2026-03-20 16:01:27.443367 | controller | openshift_obs : Wait for observability operator deployment ------------- 51.05s 2026-03-20 16:01:27.443374 | controller | run_hook : Run hook without retry - Download needed tools -------------- 47.69s 2026-03-20 16:01:27.443379 | controller | run_hook : Run hook without retry - Deploy watcher operator ------------ 42.79s 2026-03-20 16:01:27.443383 | controller | install_yamls_makes : Run download_tools ------------------------------- 39.18s 2026-03-20 16:01:27.443387 | controller | ci_setup : Install needed packages ------------------------------------- 32.91s 2026-03-20 16:01:27.443392 | controller | repo_setup : Initialize python venv and install requirements ------------ 9.05s 2026-03-20 16:01:27.443396 | controller | ci_setup : Install openshift client ------------------------------------- 4.41s 2026-03-20 16:01:27.443400 | controller | openshift_setup : Create required namespaces ---------------------------- 1.87s 2026-03-20 16:01:27.443405 | controller | Gathering Facts --------------------------------------------------------- 1.24s 2026-03-20 16:01:27.443409 | controller | repo_setup : Get repo-setup repository ---------------------------------- 1.11s 2026-03-20 16:01:27.443419 | controller | ci_setup : Manage directories ------------------------------------------- 1.06s 2026-03-20 16:01:27.443423 | controller | repo_setup : Make sure git-core package is installed -------------------- 1.00s 2026-03-20 16:01:27.443427 | controller | repo_setup : Install repo-setup package --------------------------------- 0.96s 2026-03-20 16:01:27.443432 | controller | openshift_setup : Gather network.operator info -------------------------- 0.95s 2026-03-20 16:01:27.443436 | controller | openshift_setup : Patch network operator -------------------------------- 0.94s 2026-03-20 16:01:27.443440 | controller | openshift_obs : Wait for observability-operator pod --------------------- 0.90s 2026-03-20 16:01:27.443444 | controller | openshift_obs : Install cluster observability operator. ----------------- 0.85s 2026-03-20 16:01:27.443448 | controller | openshift_setup : Patch samples registry configuration ------------------ 0.83s 2026-03-20 16:01:27.443453 | controller | Friday 20 March 2026 16:01:27 +0000 (0:02:50.033) 0:12:02.965 ********** 2026-03-20 16:01:27.443457 | controller | =============================================================================== 2026-03-20 16:01:27.443461 | controller | install_yamls_makes --------------------------------------------------- 330.12s 2026-03-20 16:01:27.443465 | controller | ansible.builtin.command ----------------------------------------------- 170.36s 2026-03-20 16:01:27.443470 | controller | run_hook --------------------------------------------------------------- 94.19s 2026-03-20 16:01:27.443474 | controller | openshift_obs ---------------------------------------------------------- 52.79s 2026-03-20 16:01:27.443478 | controller | ci_setup --------------------------------------------------------------- 39.68s 2026-03-20 16:01:27.443483 | controller | repo_setup ------------------------------------------------------------- 18.05s 2026-03-20 16:01:27.443487 | controller | openshift_setup --------------------------------------------------------- 5.13s 2026-03-20 16:01:27.443491 | controller | openshift_login --------------------------------------------------------- 4.22s 2026-03-20 16:01:27.443496 | controller | install_yamls ----------------------------------------------------------- 3.82s 2026-03-20 16:01:27.443502 | controller | cifmw_setup ------------------------------------------------------------- 1.55s 2026-03-20 16:01:27.443507 | controller | gather_facts ------------------------------------------------------------ 1.24s 2026-03-20 16:01:27.443511 | controller | discover_latest_image --------------------------------------------------- 0.61s 2026-03-20 16:01:27.443517 | controller | install_ca -------------------------------------------------------------- 0.45s 2026-03-20 16:01:27.443523 | controller | networking_mapper ------------------------------------------------------- 0.30s 2026-03-20 16:01:27.443527 | controller | ansible.builtin.set_fact ------------------------------------------------ 0.13s 2026-03-20 16:01:27.443532 | controller | ansible.builtin.assert -------------------------------------------------- 0.11s 2026-03-20 16:01:27.443536 | controller | ansible.builtin.include_role -------------------------------------------- 0.11s 2026-03-20 16:01:27.443540 | controller | ansible.builtin.include_vars -------------------------------------------- 0.05s 2026-03-20 16:01:27.443544 | controller | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2026-03-20 16:01:27.443549 | controller | total ----------------------------------------------------------------- 722.90s 2026-03-20 16:01:27.921719 | controller | changed 2026-03-20 16:01:27.999457 | 2026-03-20 16:01:27.999739 | PLAY RECAP 2026-03-20 16:01:27.999835 | controller | ok: 1 changed: 1 unreachable: 0 failed: 0 skipped: 0 rescued: 0 ignored: 0 2026-03-20 16:01:27.999881 | 2026-03-20 16:01:28.209145 | PRE-RUN END RESULT_NORMAL: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/kuttl/kuttl-from-operator-deps.yaml@main] 2026-03-20 16:01:28.220715 | RUN START: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/kuttl/kuttl-from-operator-run.yaml@main] 2026-03-20 16:01:28.865060 | 2026-03-20 16:01:28.865266 | PLAY [Run ci/playbooks/kuttl/run-kuttl-from-operator-targets.yaml] 2026-03-20 16:01:28.889270 | 2026-03-20 16:01:28.889357 | TASK [Run kuttl tests from operator playbook] 2026-03-20 16:01:29.976089 | controller | 2026-03-20 16:01:30.020184 | controller | PLAY [controller] ************************************************************** 2026-03-20 16:01:30.020256 | controller | 2026-03-20 16:01:30.020264 | controller | TASK [Run hooks before running kuttl tests name=run_hook] ********************** 2026-03-20 16:01:30.020269 | controller | Friday 20 March 2026 16:01:29 +0000 (0:00:00.045) 0:00:00.045 ********** 2026-03-20 16:01:30.020273 | controller | Friday 20 March 2026 16:01:29 +0000 (0:00:00.043) 0:00:00.043 ********** 2026-03-20 16:01:30.020286 | controller | 2026-03-20 16:01:30.088143 | controller | TASK [run_hook : Assert parameters are valid quiet=True, that=['_list_hooks is not string', '_list_hooks is not mapping', '_list_hooks is iterable', '(hooks | default([])) is not string', '(hooks | default([])) is not mapping', '(hooks | default([])) is iterable']] *** 2026-03-20 16:01:30.088214 | controller | Friday 20 March 2026 16:01:30 +0000 (0:00:00.044) 0:00:00.089 ********** 2026-03-20 16:01:30.088221 | controller | Friday 20 March 2026 16:01:30 +0000 (0:00:00.044) 0:00:00.087 ********** 2026-03-20 16:01:30.088252 | controller | ok: [controller] 2026-03-20 16:01:30.190670 | controller | 2026-03-20 16:01:30.190751 | controller | TASK [run_hook : Assert single hooks are all mappings quiet=True, that=['_not_mapping_hooks | length == 0'], msg=All single hooks must be a list of mappings or a mapping.] *** 2026-03-20 16:01:30.190769 | controller | Friday 20 March 2026 16:01:30 +0000 (0:00:00.068) 0:00:00.157 ********** 2026-03-20 16:01:30.190775 | controller | Friday 20 March 2026 16:01:30 +0000 (0:00:00.068) 0:00:00.155 ********** 2026-03-20 16:01:30.190789 | controller | ok: [controller] 2026-03-20 16:01:30.327980 | controller | 2026-03-20 16:01:30.328047 | controller | TASK [run_hook : Loop on hooks for pre_kuttl_from_operator _raw_params={{ hook.type }}.yml] *** 2026-03-20 16:01:30.328056 | controller | Friday 20 March 2026 16:01:30 +0000 (0:00:00.102) 0:00:00.259 ********** 2026-03-20 16:01:30.328063 | controller | Friday 20 March 2026 16:01:30 +0000 (0:00:00.102) 0:00:00.258 ********** 2026-03-20 16:01:30.328079 | controller | skipping: [controller] 2026-03-20 16:26:25.539943 | controller | 2026-03-20 16:26:25.540042 | controller | TASK [run kuttl test suite from operator Makefile chdir={{ ansible_user_dir }}/{{ operator_basedir }}, _raw_params=make kuttl-test-run] *** 2026-03-20 16:26:25.540049 | controller | Friday 20 March 2026 16:01:30 +0000 (0:00:00.137) 0:00:00.396 ********** 2026-03-20 16:26:25.540055 | controller | Friday 20 March 2026 16:01:30 +0000 (0:00:00.137) 0:00:00.395 ********** 2026-03-20 16:26:25.540066 | controller | task path: /home/zuul/src/github.com/openstack-k8s-operators/ci-framework/ci/playbooks/kuttl/run-kuttl-from-operator-targets.yaml:16 2026-03-20 16:26:25.540096 | controller | fatal: [controller]: FAILED! => 2026-03-20 16:26:25.540106 | controller | changed: true 2026-03-20 16:26:25.540110 | controller | cmd: 2026-03-20 16:26:25.540115 | controller | - make 2026-03-20 16:26:25.540119 | controller | - kuttl-test-run 2026-03-20 16:26:25.540123 | controller | delta: '0:24:54.158699' 2026-03-20 16:26:25.540132 | controller | end: '2026-03-20 16:26:24.849894' 2026-03-20 16:26:25.540137 | controller | msg: non-zero return code 2026-03-20 16:26:25.540141 | controller | rc: 2 2026-03-20 16:26:25.540146 | controller | start: '2026-03-20 16:01:30.691195' 2026-03-20 16:26:25.540150 | controller | stderr: |- 2026-03-20 16:26:25.540155 | controller | Warning: spec.privateKey.rotationPolicy: In cert-manager >= v1.18.0, the default value changed from `Never` to `Always`. 2026-03-20 16:26:25.540160 | controller | Warning: The 'cinderBackup' field is deprecated and will be removed in a future release. Please migrate to 'cinderBackups'. 2026-03-20 16:26:25.540164 | controller | make: *** [Makefile:444: kuttl-test-run] Error 1 2026-03-20 16:26:25.540169 | controller | stderr_lines: 2026-03-20 16:26:25.540173 | controller | - 'Warning: spec.privateKey.rotationPolicy: In cert-manager >= v1.18.0, the default 2026-03-20 16:26:25.540178 | controller | value changed from `Never` to `Always`.' 2026-03-20 16:26:25.540182 | controller | - 'Warning: The ''cinderBackup'' field is deprecated and will be removed in a future 2026-03-20 16:26:25.540187 | controller | release. Please migrate to ''cinderBackups''.' 2026-03-20 16:26:25.540191 | controller | - 'make: *** [Makefile:444: kuttl-test-run] Error 1' 2026-03-20 16:26:25.540219 | controller | stdout: "oc kuttl test --v 1 --start-kind=false --config test/kuttl/test-suites/default/config.yaml\n=== 2026-03-20 16:26:25.540224 | controller | RUN kuttl\n harness.go:463: starting setup\n harness.go:255: running tests 2026-03-20 16:26:25.540228 | controller | using configured kubeconfig.\n harness.go:278: Successful connection to cluster 2026-03-20 16:26:25.540233 | controller | at: https://api.crc.testing:6443\n harness.go:363: running tests\n harness.go:75: 2026-03-20 16:26:25.540238 | controller | going to run test suite with timeout of 300 seconds for each step\n harness.go:375: 2026-03-20 16:26:25.540243 | controller | testsuite: test/kuttl/test-suites/default/ has 11 tests\n=== RUN kuttl/harness\n=== 2026-03-20 16:26:25.540247 | controller | RUN kuttl/harness/appcred-tests\n=== PAUSE kuttl/harness/appcred-tests\n=== RUN 2026-03-20 16:26:25.540251 | controller | \ kuttl/harness/common\n=== PAUSE kuttl/harness/common\n=== RUN kuttl/harness/deps\n=== 2026-03-20 16:26:25.540256 | controller | PAUSE kuttl/harness/deps\n=== RUN kuttl/harness/watcher\n=== PAUSE kuttl/harness/watcher\n=== 2026-03-20 16:26:25.540260 | controller | RUN kuttl/harness/watcher-api-scaling\n=== PAUSE kuttl/harness/watcher-api-scaling\n=== 2026-03-20 16:26:25.540264 | controller | RUN kuttl/harness/watcher-cinder\n=== PAUSE kuttl/harness/watcher-cinder\n=== 2026-03-20 16:26:25.540269 | controller | RUN kuttl/harness/watcher-notification\n=== PAUSE kuttl/harness/watcher-notification\n=== 2026-03-20 16:26:25.540273 | controller | RUN kuttl/harness/watcher-rmquser\n=== PAUSE kuttl/harness/watcher-rmquser\n=== 2026-03-20 16:26:25.540278 | controller | RUN kuttl/harness/watcher-tls\n=== PAUSE kuttl/harness/watcher-tls\n=== RUN kuttl/harness/watcher-tls-certs-change\n=== 2026-03-20 16:26:25.540282 | controller | PAUSE kuttl/harness/watcher-tls-certs-change\n=== RUN kuttl/harness/watcher-topology\n=== 2026-03-20 16:26:25.540295 | controller | PAUSE kuttl/harness/watcher-topology\n=== CONT kuttl/harness/appcred-tests\n logger.go:42: 2026-03-20 16:26:25.540299 | controller | 16:01:30 | appcred-tests | Skipping creation of user-supplied namespace: watcher-kuttl-default\n 2026-03-20 16:26:25.540303 | controller | \ logger.go:42: 16:01:30 | appcred-tests/0-cleanup | starting test step 0-cleanup\n 2026-03-20 16:26:25.540311 | controller | \ logger.go:42: 16:01:30 | appcred-tests/0-cleanup | test step completed 0-cleanup\n 2026-03-20 16:26:25.540316 | controller | \ logger.go:42: 16:01:30 | appcred-tests/1-deploy | starting test step 1-deploy\n 2026-03-20 16:26:25.540320 | controller | \ logger.go:42: 16:01:31 | appcred-tests/1-deploy | Watcher:watcher-kuttl-default/watcher-kuttl 2026-03-20 16:26:25.540324 | controller | created\n logger.go:42: 16:01:31 | appcred-tests/1-deploy | running command: 2026-03-20 16:26:25.540329 | controller | [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n oc wait -n \"${NS}\" 2026-03-20 16:26:25.540333 | controller | watcher watcher-kuttl --for=condition=Ready --timeout=600s]\n logger.go:42: 16:02:31 2026-03-20 16:26:25.540337 | controller | | appcred-tests/1-deploy | watcher.watcher.openstack.org/watcher-kuttl condition 2026-03-20 16:26:25.540342 | controller | met\n logger.go:42: 16:02:31 | appcred-tests/1-deploy | test step completed 1-deploy\n 2026-03-20 16:26:25.540346 | controller | \ logger.go:42: 16:02:31 | appcred-tests/2-deploy-appcred | starting test step 2026-03-20 16:26:25.540350 | controller | 2-deploy-appcred\n logger.go:42: 16:02:31 | appcred-tests/2-deploy-appcred | 2026-03-20 16:26:25.540354 | controller | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.540366 | controller | \ old_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.540370 | controller | \ old_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.540375 | controller | \ old_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.540379 | controller | jsonpath='{.metadata.uid}')\n oc create configmap appcred-watcher-pre \\\n 2026-03-20 16:26:25.540389 | controller | \ --from-literal=api_uid=\"${old_api_uid}\" \\\n --from-literal=applier_uid=\"${old_applier_uid}\" 2026-03-20 16:26:25.540394 | controller | \\\n --from-literal=decision_engine_uid=\"${old_de_uid}\" \\\n --dry-run=client 2026-03-20 16:26:25.540398 | controller | -o yaml | oc apply -n \"${NS}\" -f -\n \n cat <<'EOF' | oc apply -n 2026-03-20 16:26:25.540403 | controller | \"${NS}\" -f -\n apiVersion: keystone.openstack.org/v1beta1\n kind: 2026-03-20 16:26:25.540407 | controller | KeystoneApplicationCredential\n metadata:\n name: ac-watcher\n spec:\n 2026-03-20 16:26:25.540411 | controller | \ secret: osp-secret\n passwordSelector: WatcherPassword\n userName: 2026-03-20 16:26:25.540415 | controller | watcher\n roles:\n - admin\n - service\n unrestricted: 2026-03-20 16:26:25.540420 | controller | false\n EOF\n \n oc patch watcher watcher-kuttl -n \"${NS}\" 2026-03-20 16:26:25.540424 | controller | --type=merge -p '{\"spec\":{\"auth\":{\"applicationCredentialSecret\":\"ac-watcher-secret\"}}}']\n 2026-03-20 16:26:25.540428 | controller | \ logger.go:42: 16:02:32 | appcred-tests/2-deploy-appcred | configmap/appcred-watcher-pre 2026-03-20 16:26:25.540433 | controller | created\n logger.go:42: 16:02:32 | appcred-tests/2-deploy-appcred | keystoneapplicationcredential.keystone.openstack.org/ac-watcher 2026-03-20 16:26:25.540437 | controller | created\n logger.go:42: 16:02:32 | appcred-tests/2-deploy-appcred | watcher.watcher.openstack.org/watcher-kuttl 2026-03-20 16:26:25.540441 | controller | patched\n logger.go:42: 16:02:32 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.540446 | controller | [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n echo \"Waiting 2026-03-20 16:26:25.540450 | controller | for KeystoneApplicationCredential ac-watcher to be Ready...\"\n oc wait -n 2026-03-20 16:26:25.540454 | controller | \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n \n ac_id=$(oc 2026-03-20 16:26:25.540459 | controller | get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n if [ 2026-03-20 16:26:25.540464 | controller | -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.540469 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.540473 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.540478 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.540482 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.540487 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.540491 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.540495 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.540499 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.540503 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.540510 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.540514 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.540519 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.540523 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.540527 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.540534 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.540538 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.540542 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.540547 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.540551 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.540555 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.540559 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.540564 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.540568 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.540574 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.540579 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.540583 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.540587 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.540592 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.540596 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.540600 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.540604 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.540609 | controller | expected application_credential_id\"]\n logger.go:42: 16:02:32 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.540613 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.540617 | controller | 16:07:32 | appcred-tests/2-deploy-appcred | error: timed out waiting for the condition 2026-03-20 16:26:25.540621 | controller | on keystoneapplicationcredentials/ac-watcher\n logger.go:42: 16:07:33 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.540626 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.540630 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.540634 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.540638 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.540643 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.540647 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.540651 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.540655 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.540660 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.540664 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.540668 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.540673 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.540680 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.540685 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.540689 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.540719 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.540724 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.540728 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.540733 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.540737 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.540741 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.540745 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.540750 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.540754 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.540758 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.540763 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.540767 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.540771 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.540775 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.540788 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.540795 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.540799 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.540804 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.540808 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.540812 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.540816 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.540821 | controller | expected application_credential_id\"]\n logger.go:42: 16:07:33 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.540825 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.540829 | controller | 16:07:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.540834 | controller | \"ac-watcher\" not found\n logger.go:42: 16:07:34 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.540838 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.540842 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.540846 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.540853 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.540857 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.540862 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.540866 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.540870 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.540874 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.540879 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.540883 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.540887 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.540891 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.540896 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.540900 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.540904 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.540908 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.540913 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.540917 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.540921 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.540926 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.540930 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.540934 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.540938 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.540943 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.540947 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.540951 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.540955 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.540960 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.540966 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.540970 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.540976 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.540981 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.540985 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.540991 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.540995 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.541002 | controller | expected application_credential_id\"]\n logger.go:42: 16:07:34 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.541006 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.541010 | controller | 16:07:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.541015 | controller | \"ac-watcher\" not found\n logger.go:42: 16:07:35 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.541019 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.541023 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.541027 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.541032 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.541036 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.541040 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.541045 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.541049 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.541053 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.541057 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.541062 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.541066 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.541070 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.541074 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.541079 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.541083 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.541087 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.541091 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.541096 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.541101 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.541105 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.541109 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.541114 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.541118 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.541122 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.541126 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.541131 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.541135 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.541141 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.541146 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.541150 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.541154 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.541160 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.541165 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.541169 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.541173 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.541178 | controller | expected application_credential_id\"]\n logger.go:42: 16:07:35 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.541182 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.541186 | controller | 16:07:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.541190 | controller | \"ac-watcher\" not found\n logger.go:42: 16:07:37 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.541195 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.541199 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.541203 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.541207 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.541211 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.541216 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.541220 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.541224 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.541228 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.541233 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.541237 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.541243 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.541247 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.541251 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.541256 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.541260 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.541264 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.541268 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.541273 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.541279 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.541283 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.541288 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.541292 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.541296 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.541301 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.541305 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.541309 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.541313 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.541317 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.541322 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.541326 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.541330 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.541334 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.541339 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.541345 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.541349 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.541354 | controller | expected application_credential_id\"]\n logger.go:42: 16:07:37 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.541359 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.541363 | controller | 16:07:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.541368 | controller | \"ac-watcher\" not found\n logger.go:42: 16:07:38 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.541372 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.541376 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.541380 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.541388 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.541392 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.541397 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.541401 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.541405 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.541409 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.541414 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.541418 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.541422 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.541428 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.541433 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.541437 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.541441 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.541445 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.541450 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.541454 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.541458 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.541463 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.541467 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.541471 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.541475 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.541480 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.541484 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.541488 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.541492 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.541497 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.541501 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.541505 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.541511 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.541515 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.541520 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.541524 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.541530 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.541535 | controller | expected application_credential_id\"]\n logger.go:42: 16:07:38 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.541539 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.541543 | controller | 16:07:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.541548 | controller | \"ac-watcher\" not found\n logger.go:42: 16:07:39 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.541552 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.541556 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.541560 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.541567 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.541571 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.541576 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.541580 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.541584 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.541588 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.541592 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.541597 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.541601 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.541605 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.541609 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.541614 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.541618 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.541622 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.541628 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.541632 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.541636 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.541641 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.541645 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.541649 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.541654 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.541658 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.541662 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.541666 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.541671 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.541675 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.541679 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.541683 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.541688 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.541707 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.541713 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.541718 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.541722 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.541729 | controller | expected application_credential_id\"]\n logger.go:42: 16:07:39 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.541736 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.541740 | controller | 16:07:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.541744 | controller | \"ac-watcher\" not found\n logger.go:42: 16:07:40 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.541749 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.541753 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.541757 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.541761 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.541766 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.541770 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.541774 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.541778 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.541783 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.541787 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.541793 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.541798 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.541802 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.541806 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.541812 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.541816 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.541821 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.541825 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.541829 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.541834 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.541838 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.541842 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.541846 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.541851 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.541855 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.541859 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.541863 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.541868 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.541874 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.541878 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.541882 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.541887 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.541891 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.541895 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.541900 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.541904 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.541908 | controller | expected application_credential_id\"]\n logger.go:42: 16:07:40 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.541912 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.541918 | controller | 16:07:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.541923 | controller | \"ac-watcher\" not found\n logger.go:42: 16:07:41 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.541927 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.541931 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.541935 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.541940 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.541944 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.541948 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.541952 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.541957 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.541961 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.541965 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.541969 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.541974 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.541978 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.541982 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.541986 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.541991 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.541995 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.541999 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.542004 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.542010 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.542015 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.542019 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.542023 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.542028 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.542032 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.542036 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.542040 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.542047 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.542051 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.542055 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.542060 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.542064 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.542068 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.542072 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.542077 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.542081 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.542085 | controller | expected application_credential_id\"]\n logger.go:42: 16:07:41 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.542089 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.542094 | controller | 16:07:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.542100 | controller | \"ac-watcher\" not found\n logger.go:42: 16:07:42 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.542104 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.542108 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.542113 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.542117 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.542121 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.542125 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.542130 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.542134 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.542138 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.542142 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.542147 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.542151 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.542157 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.542162 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.542166 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.542170 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.542174 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.542179 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.542183 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.542187 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.542192 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.542196 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.542200 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.542204 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.542208 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.542213 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.542217 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.542221 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.542225 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.542391 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.542395 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.542399 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.542404 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.542408 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.542412 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.542416 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.542421 | controller | expected application_credential_id\"]\n logger.go:42: 16:07:42 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.542425 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.542429 | controller | 16:07:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.542434 | controller | \"ac-watcher\" not found\n logger.go:42: 16:07:43 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.542438 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.542444 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.542448 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.542455 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.542461 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.542475 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.542479 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.542484 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.542488 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.542492 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.542497 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.542501 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.542505 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.542509 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.542514 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.542520 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.542524 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.542528 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.542532 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.542537 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.542551 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.542556 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.542560 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.542564 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.542569 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.542573 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.542577 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.542581 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.542588 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.542593 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.542598 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.542603 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.542661 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.542666 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.542670 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.542675 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.542685 | controller | expected application_credential_id\"]\n logger.go:42: 16:07:43 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.542690 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.542734 | controller | 16:07:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.542739 | controller | \"ac-watcher\" not found\n logger.go:42: 16:07:45 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.542744 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.542748 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.542760 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.542765 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.542769 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.542773 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.542778 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.542782 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.542786 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.542790 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.542795 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.542799 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.542803 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.542807 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.542812 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.542816 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.542820 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.542824 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.542829 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.542833 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.542837 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.542842 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.542846 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.542850 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.542854 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.542859 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.542866 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.542870 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.542876 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.542880 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.542884 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.542889 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.542893 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.542897 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.542901 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.542906 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.542910 | controller | expected application_credential_id\"]\n logger.go:42: 16:07:45 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.542914 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.542918 | controller | 16:07:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.542923 | controller | \"ac-watcher\" not found\n logger.go:42: 16:07:46 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.542927 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.542931 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.542935 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.542948 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.542952 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.542957 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.542961 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.542965 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.542970 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.542974 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.542978 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.542982 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.542986 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.542999 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.543003 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.543007 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.543012 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.543016 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.543020 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.543026 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.543030 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.543035 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.543039 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.543043 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.543047 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.543052 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.543056 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.543060 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.543064 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.543068 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.543073 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.543077 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.543081 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.543085 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.543090 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.543094 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.543098 | controller | expected application_credential_id\"]\n logger.go:42: 16:07:46 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.543102 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.543106 | controller | 16:07:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.543111 | controller | \"ac-watcher\" not found\n logger.go:42: 16:07:47 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.543115 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.543119 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.543123 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.543127 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.543132 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.543143 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.543147 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.543152 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.543158 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.543162 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.543166 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.543171 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.543176 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.543181 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.543185 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.543189 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.543193 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.543198 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.543202 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.543206 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.543210 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.543214 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.543219 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.543223 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.543227 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.543231 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.543235 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.543240 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.543244 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.543248 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.543252 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.543256 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.543261 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.543265 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.543269 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.543273 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.543278 | controller | expected application_credential_id\"]\n logger.go:42: 16:07:47 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.543282 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.543286 | controller | 16:07:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.543291 | controller | \"ac-watcher\" not found\n logger.go:42: 16:07:48 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.543295 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.543299 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.543303 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.543309 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.543314 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.543318 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.543328 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.543333 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.543337 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.543341 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.543345 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.543350 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.543354 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.543358 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.543364 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.543368 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.543372 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.543376 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.543381 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.543385 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.543389 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.543393 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.543397 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.543402 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.543407 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.543411 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.543416 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.543420 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.543424 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.543449 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.543454 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.543458 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.543462 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.543466 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.543471 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.543475 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.543481 | controller | expected application_credential_id\"]\n logger.go:42: 16:07:48 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.543485 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.543489 | controller | 16:07:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.543494 | controller | \"ac-watcher\" not found\n logger.go:42: 16:07:49 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.543498 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.543502 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.543506 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.543511 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.543515 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.543519 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.543524 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.543535 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.543539 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.543543 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.543548 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.543552 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.543556 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.543560 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.543565 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.543569 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.543573 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.543577 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.543581 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.543587 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.543591 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.543595 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.543599 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.543604 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.543608 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.543612 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.543616 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.543620 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.543626 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.543631 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.543635 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.543639 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.543643 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.543648 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.543652 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.543656 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.543660 | controller | expected application_credential_id\"]\n logger.go:42: 16:07:49 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.543664 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.543669 | controller | 16:07:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.543673 | controller | \"ac-watcher\" not found\n logger.go:42: 16:07:50 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.543677 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.543681 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.543686 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.543690 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.543712 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.543717 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.543721 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.543726 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.543730 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.543742 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.543746 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.543751 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.543755 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.543760 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.543764 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.543768 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.543772 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.543777 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.543781 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.543787 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.543791 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.543796 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.543800 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.543804 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.543808 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.543813 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.543817 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.543821 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.543825 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.543831 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.543835 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.543839 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.543843 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.543848 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.543852 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.543856 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.543860 | controller | expected application_credential_id\"]\n logger.go:42: 16:07:50 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.543865 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.543869 | controller | 16:07:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.543873 | controller | \"ac-watcher\" not found\n logger.go:42: 16:07:51 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.543877 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.543881 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.543886 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.543890 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.543894 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.543898 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.543902 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.543907 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.543911 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.543915 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.543926 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.543930 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.543936 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.543940 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.543944 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.543949 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.543953 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.543957 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.543961 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.543966 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.543970 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.543974 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.543978 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.543983 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.543988 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.543992 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.543996 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.544000 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.544005 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.544009 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.544013 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.544017 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.544021 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.544026 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.544030 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.544034 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.544038 | controller | expected application_credential_id\"]\n logger.go:42: 16:07:51 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.544042 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.544046 | controller | 16:07:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.544051 | controller | \"ac-watcher\" not found\n logger.go:42: 16:07:53 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.544055 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.544059 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.544064 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.544070 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.544074 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.544078 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.544082 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.544087 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.544091 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.544095 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.544099 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.544103 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.544114 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.544119 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.544123 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.544128 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.544140 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.544144 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.544149 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.544153 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.544157 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.544162 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.544166 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.544170 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.544174 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.544178 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.544183 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.544187 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.544191 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.544196 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.544200 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.544204 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.544208 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.544220 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.544224 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.544228 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.544236 | controller | expected application_credential_id\"]\n logger.go:42: 16:07:53 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.544240 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.544245 | controller | 16:07:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.544249 | controller | \"ac-watcher\" not found\n logger.go:42: 16:07:54 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.544253 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.544257 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.544263 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.544268 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.544272 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.544277 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.544281 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.544285 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.544297 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.544301 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.544306 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.544310 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.544314 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.544324 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.544329 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.544333 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.544337 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.544342 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.544346 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.544350 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.544354 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.544359 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.544370 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.544375 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.544379 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.544383 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.544388 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.544392 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.544398 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.544403 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.544407 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.544411 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.544415 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.544420 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.544424 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.544428 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.544433 | controller | expected application_credential_id\"]\n logger.go:42: 16:07:54 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.544437 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.544441 | controller | 16:07:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.544446 | controller | \"ac-watcher\" not found\n logger.go:42: 16:07:55 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.544450 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.544454 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.544458 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.544463 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.544467 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.544471 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.544475 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.544480 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.544484 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.544488 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.544492 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.544497 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.544501 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.544505 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.544509 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.544520 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.544524 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.544529 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.544533 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.544540 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.544544 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.544549 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.544554 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.544558 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.544562 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.544567 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.544571 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.544575 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.544579 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.544584 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.544588 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.544592 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.544596 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.544601 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.544605 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.544609 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.544613 | controller | expected application_credential_id\"]\n logger.go:42: 16:07:55 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.544618 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.544622 | controller | 16:07:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.544626 | controller | \"ac-watcher\" not found\n logger.go:42: 16:07:56 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.544631 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.544635 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.544639 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.544643 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.544648 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.544652 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.544656 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.544660 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.544665 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.544669 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.544673 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.544677 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.544683 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.544688 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.544709 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.544716 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.544726 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.544731 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.544735 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.544740 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.544744 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.544748 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.544753 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.544757 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.544761 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.544765 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.544770 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.544774 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.544778 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.544782 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.544787 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.544791 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.544795 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.544800 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.544804 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.544808 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.544812 | controller | expected application_credential_id\"]\n logger.go:42: 16:07:56 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.544818 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.544823 | controller | 16:07:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.544827 | controller | \"ac-watcher\" not found\n logger.go:42: 16:07:57 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.544833 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.544838 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.544852 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.544858 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.544862 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.544867 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.544871 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.544876 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.544880 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.544884 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.544888 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.544893 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.544897 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.544901 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.544906 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.544910 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.544914 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.544924 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.544929 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.544933 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.544938 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.544942 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.544947 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.544951 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.544955 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.544959 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.544963 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.544968 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.544972 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.544976 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.544980 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.544985 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.544989 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.544993 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.544997 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.545002 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.545008 | controller | expected application_credential_id\"]\n logger.go:42: 16:07:57 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.545012 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.545016 | controller | 16:07:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.545021 | controller | \"ac-watcher\" not found\n logger.go:42: 16:07:58 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.545025 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.545029 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.545033 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.545038 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.545042 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.545046 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.545050 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.545055 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.545059 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.545063 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.545067 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.545072 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.545077 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.545081 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.545086 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.545090 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.545094 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.545098 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.545110 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.545114 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.545119 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.545123 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.545127 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.545131 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.545136 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.545140 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.545144 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.545148 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.545155 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.545161 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.545165 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.545169 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.545173 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.545178 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.545182 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.545186 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.545190 | controller | expected application_credential_id\"]\n logger.go:42: 16:07:58 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.545194 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.545199 | controller | 16:07:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.545203 | controller | \"ac-watcher\" not found\n logger.go:42: 16:07:59 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.545207 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.545211 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.545216 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.545220 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.545224 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.545228 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.545233 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.545237 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.545241 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.545245 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.545249 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.545254 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.545258 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.545262 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.545266 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.545271 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.545275 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.545279 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.545283 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.545289 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.545299 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.545304 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.545308 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.545312 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.545317 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.545321 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.545333 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.545337 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.545341 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.545345 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.545350 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.545354 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.545358 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.545362 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.545366 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.545370 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.545375 | controller | expected application_credential_id\"]\n logger.go:42: 16:07:59 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.545379 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.545383 | controller | 16:08:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.545388 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:01 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.545393 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.545397 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.545401 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.545406 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.545410 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.545414 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.545418 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.545422 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.545427 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.545431 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.545435 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.545439 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.545445 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.545449 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.545453 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.545458 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.545462 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.545466 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.545470 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.545475 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.545480 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.545490 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.545495 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.545499 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.545503 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.545507 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.545512 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.545516 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.545520 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.545524 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.545528 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.545532 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.545537 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.545541 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.545545 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.545549 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.545553 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:01 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.545557 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.545561 | controller | 16:08:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.545566 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:02 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.545570 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.545574 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.545578 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.545585 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.545589 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.545593 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.545598 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.545602 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.545606 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.545610 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.545614 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.545618 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.545622 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.545626 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.545630 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.545636 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.545640 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.545644 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.545648 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.545652 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.545656 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.545661 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.545665 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.545675 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.545679 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.545683 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.545688 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.545704 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.545710 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.545715 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.545719 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.545723 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.545727 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.545731 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.545735 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.545740 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.545746 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:02 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.545750 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.545754 | controller | 16:08:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.545758 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:03 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.545762 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.545766 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.545770 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.545775 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.545779 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.545783 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.545787 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.545791 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.545796 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.545800 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.545804 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.545808 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.545812 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.545816 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.545828 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.545833 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.545837 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.545841 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.545845 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.545849 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.545854 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.545858 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.545862 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.545866 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.545876 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.545881 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.545885 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.545889 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.545895 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.545899 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.545904 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.545908 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.545912 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.545916 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.545920 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.545924 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.545929 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:03 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.545933 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.545937 | controller | 16:08:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.545943 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:04 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.545954 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.545958 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.545962 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.545967 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.545971 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.545975 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.545979 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.545983 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.545987 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.545992 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.545996 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.546000 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.546004 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.546008 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.546013 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.546025 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.546029 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.546034 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.546038 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.546044 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.546048 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.546052 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.546056 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.546060 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.546065 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.546069 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.546079 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.546083 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.546088 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.546092 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.546096 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.546100 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.546104 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.546109 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.546113 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.546117 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.546121 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:04 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.546125 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.546129 | controller | 16:08:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.546134 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:05 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.546138 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.546142 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.546146 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.546150 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.546154 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.546159 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.546163 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.546167 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.546171 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.546175 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.546179 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.546183 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.546189 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.546194 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.546199 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.546203 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.546207 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.546211 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.546215 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.546220 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.546224 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.546228 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.546232 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.546236 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.546240 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.546244 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.546249 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.546259 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.546263 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.546267 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.546271 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.546276 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.546280 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.546292 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.546296 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.546300 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.546304 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:05 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.546308 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.546312 | controller | 16:08:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.546317 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:06 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.546321 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.546325 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.546329 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.546336 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.546340 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.546345 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.546349 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.546353 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.546357 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.546361 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.546365 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.546369 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.546374 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.546378 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.546382 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.546386 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.546390 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.546394 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.546399 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.546404 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.546408 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.546412 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.546416 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.546421 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.546425 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.546429 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.546433 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.546437 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.546449 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.546453 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.546458 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.546462 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.546466 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.546470 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.546474 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.546478 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.546484 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:06 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.546489 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.546494 | controller | 16:08:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.546498 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:08 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.546502 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.546506 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.546510 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.546514 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.546519 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.546523 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.546527 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.546531 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.546535 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.546539 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.546543 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.546548 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.546552 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.546556 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.546560 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.546564 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.546568 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.546572 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.546577 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.546581 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.546585 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.546589 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.546593 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.546597 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.546601 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.546606 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.546610 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.546614 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.546620 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.546624 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.546634 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.546639 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.546643 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.546647 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.546651 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.546656 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.546660 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:08 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.546664 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.546668 | controller | 16:08:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.546672 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:09 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.546676 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.546681 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.546685 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.546689 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.546709 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.546714 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.546718 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.546722 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.546726 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.546731 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.546735 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.546739 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.546744 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.546748 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.546752 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.546757 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.546761 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.546765 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.546769 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.546775 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.546779 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.546783 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.546788 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.546792 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.546796 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.546800 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.546804 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.546808 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.546812 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.546817 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.546821 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.546831 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.546836 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.546840 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.546844 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.546848 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.546852 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:09 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.546856 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.546860 | controller | 16:08:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.546865 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:10 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.546869 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.546873 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.546877 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.546882 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.546888 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.546892 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.546896 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.546900 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.546904 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.546909 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.546913 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.546917 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.546923 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.546927 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.546931 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.546935 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.546939 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.546944 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.546948 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.546952 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.546956 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.546960 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.546965 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.546969 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.546973 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.546977 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.546981 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.546985 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.546990 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.546994 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.546998 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.547002 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.547012 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.547017 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.547022 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.547027 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.547031 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:10 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.547035 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.547039 | controller | 16:08:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.547043 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:11 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.547048 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.547052 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.547056 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.547062 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.547066 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.547070 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.547074 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.547078 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.547082 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.547086 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.547091 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.547095 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.547099 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.547103 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.547107 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.547111 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.547115 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.547120 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.547124 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.547128 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.547132 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.547136 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.547140 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.547145 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.547149 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.547153 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.547157 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.547161 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.547165 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.547169 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.547174 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.547178 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.547182 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.547186 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.547196 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.547201 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.547207 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:11 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.547211 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.547215 | controller | 16:08:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.547219 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:12 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.547223 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.547227 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.547231 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.547236 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.547240 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.547244 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.547248 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.547252 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.547256 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.547260 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.547264 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.547270 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.547274 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.547278 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.547283 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.547287 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.547291 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.547296 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.547300 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.547305 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.547309 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.547313 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.547317 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.547321 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.547325 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.547330 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.547334 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.547338 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.547344 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.547348 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.547352 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.547356 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.547360 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.547364 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.547369 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.547380 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.547384 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:12 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.547388 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.547392 | controller | 16:08:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.547397 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:13 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.547401 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.547405 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.547409 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.547413 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.547417 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.547421 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.547426 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.547430 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.547445 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.547449 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.547454 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.547458 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.547462 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.547466 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.547470 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.547474 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.547479 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.547483 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.547487 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.547493 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.547497 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.547501 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.547505 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.547509 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.547513 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.547518 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.547522 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.547526 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.547530 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.547534 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.547538 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.547544 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.547548 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.547552 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.547557 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.547561 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.547571 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:13 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.547576 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.547580 | controller | 16:08:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.547584 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:15 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.547588 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.547592 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.547597 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.547601 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.547605 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.547609 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.547613 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.547617 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.547622 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.547626 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.547630 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.547634 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.547642 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.547646 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.547651 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.547655 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.547659 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.547663 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.547667 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.547671 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.547676 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.547680 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.547684 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.547688 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.547703 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.547709 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.547714 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.547718 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.547722 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.547727 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.547731 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.547736 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.547740 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.547744 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.547748 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.547752 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.547756 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:15 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.547760 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.547771 | controller | 16:08:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.547775 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:16 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.547779 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.547783 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.547788 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.547794 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.547798 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.547802 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.547806 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.547810 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.547814 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.547818 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.547824 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.547828 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.547832 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.547836 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.547840 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.547845 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.547849 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.547853 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.547857 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.547861 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.547865 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.547869 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.547874 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.547878 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.547882 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.547886 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.547890 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.547894 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.547899 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.547903 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.547907 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.547911 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.547915 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.547920 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.547924 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.547928 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.547933 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:16 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.547938 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.547942 | controller | 16:08:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.547951 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:17 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.547956 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.547960 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.547964 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.547968 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.547972 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.547977 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.547981 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.547985 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.547989 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.547993 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.547997 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.548001 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.548005 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.548010 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.548014 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.548018 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.548022 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.548026 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.548030 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.548034 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.548038 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.548043 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.548047 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.548051 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.548055 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.548059 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.548063 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.548068 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.548074 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.548078 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.548082 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.548087 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.548091 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.548095 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.548099 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.548103 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.548107 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:17 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.548111 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.548116 | controller | 16:08:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.548120 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:18 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.548130 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.548135 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.548139 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.548154 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.548159 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.548163 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.548167 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.548171 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.548175 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.548179 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.548184 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.548188 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.548192 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.548196 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.548200 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.548204 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.548208 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.548213 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.548217 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.548225 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.548229 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.548234 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.548238 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.548242 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.548246 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.548250 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.548254 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.548259 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.548263 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.548267 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.548271 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.548275 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.548279 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.548283 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.548288 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.548292 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.548296 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:18 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.548300 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.548304 | controller | 16:08:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.548308 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:19 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.548312 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.548317 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.548327 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.548331 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.548335 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.548340 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.548344 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.548348 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.548352 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.548358 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.548362 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.548366 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.548372 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.548376 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.548380 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.548384 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.548389 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.548393 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.548397 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.548401 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.548405 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.548409 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.548414 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.548418 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.548422 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.548426 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.548430 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.548434 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.548439 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.548443 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.548447 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.548451 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.548455 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.548459 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.548464 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.548468 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.548472 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:19 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.548476 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.548480 | controller | 16:08:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.548484 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:20 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.548489 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.548493 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.548497 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.548508 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.548512 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.548516 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.548521 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.548525 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.548529 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.548533 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.548537 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.548542 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.548546 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.548550 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.548556 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.548560 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.548564 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.548568 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.548573 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.548577 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.548581 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.548585 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.548589 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.548593 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.548597 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.548602 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.548607 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.548611 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.548616 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.548620 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.548624 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.548628 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.548632 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.548636 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.548640 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.548645 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.548650 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:20 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.548654 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.548659 | controller | 16:08:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.548663 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:21 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.548667 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.548671 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.548675 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.548679 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.548684 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.548705 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.548711 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.548716 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.548720 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.548724 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.548728 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.548732 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.548736 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.548741 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.548745 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.548749 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.548753 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.548757 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.548761 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.548766 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.548770 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.548774 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.548778 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.548782 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.548787 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.548791 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.548795 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.548799 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.548805 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.548809 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.548813 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.548817 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.548821 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.548826 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.548830 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.548834 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.548838 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:21 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.548842 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.548846 | controller | 16:08:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.548851 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:23 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.548855 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.548859 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.548863 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.548867 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.548871 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.548876 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.548886 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.548890 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.548897 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.548901 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.548905 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.548909 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.548913 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.548917 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.548922 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.548926 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.548930 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.548934 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.548938 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.548944 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.548948 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.548953 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.548957 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.548961 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.548965 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.548969 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.548973 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.548978 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.548982 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.548987 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.548992 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.548996 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.549000 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.549004 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.549008 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.549013 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.549017 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:23 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.549022 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.549027 | controller | 16:08:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.549031 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:24 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.549035 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.549039 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.549043 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.549047 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.549052 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.549056 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.549060 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.549070 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.549074 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.549079 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.549083 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.549087 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.549093 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.549097 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.549101 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.549105 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.549109 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.549113 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.549118 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.549124 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.549128 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.549132 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.549137 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.549141 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.549148 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.549152 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.549156 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.549160 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.549165 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.549169 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.549173 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.549177 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.549181 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.549186 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.549190 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.549194 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.549198 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:24 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.549202 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.549207 | controller | 16:08:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.549211 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:25 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.549215 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.549219 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.549223 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.549229 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.549233 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.549238 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.549242 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.549246 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.549250 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.549260 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.549265 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.549269 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.549273 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.549278 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.549282 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.549286 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.549290 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.549294 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.549298 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.549312 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.549316 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.549320 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.549325 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.549329 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.549333 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.549337 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.549341 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.549345 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.549350 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.549354 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.549358 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.549362 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.549366 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.549371 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.549375 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.549379 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.549385 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:25 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.549389 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.549393 | controller | 16:08:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.549405 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:26 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.549409 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.549413 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.549418 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.549423 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.549428 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.549432 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.549436 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.549442 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.549446 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.549450 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.549461 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.549465 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.549470 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.549481 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.549485 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.549489 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.549494 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.549498 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.549502 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.549506 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.549510 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.549515 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.549519 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.549523 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.549527 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.549532 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.549536 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.549540 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.549546 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.549550 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.549554 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.549558 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.549563 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.549567 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.549571 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.549575 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.549580 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:26 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.549586 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.549591 | controller | 16:08:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.549597 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:27 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.549603 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.549608 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.549613 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.549619 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.549627 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.549633 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.549638 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.549644 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.549649 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.549654 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.549660 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.549666 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.549681 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.549689 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.549747 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.549755 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.549761 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.549768 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.549773 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.549782 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.549806 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.549813 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.549819 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.549829 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.549835 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.549841 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.549847 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.549853 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.549859 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.549865 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.549871 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.549877 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.549884 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.549889 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.549895 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.549901 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.549907 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:27 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.549912 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.549918 | controller | 16:08:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.549924 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:28 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.549930 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.549936 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.549941 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.549947 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.549953 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.549960 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.549965 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.549971 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.549977 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.549983 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.549989 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.549994 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.550008 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.550026 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.550037 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.550045 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.550052 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.550058 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.550063 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.550069 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.550075 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.550081 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.550087 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.550093 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.550100 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.550106 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.550111 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.550118 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.550124 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.550129 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.550136 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.550142 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.550148 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.550154 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.550161 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.550167 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.550173 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:28 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.550179 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.550185 | controller | 16:08:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.550191 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:29 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.550197 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.550203 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.550212 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.550221 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.550227 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.550233 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.550239 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.550245 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.550252 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.550257 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.550263 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.550269 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.550274 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.550280 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.550285 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.550301 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.550307 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.550313 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.550319 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.550327 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.550333 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.550339 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.550345 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.550350 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.550354 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.550358 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.550363 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.550367 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.550371 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.550375 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.550380 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.550384 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.550388 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.550393 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.550397 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.550401 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.550407 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:29 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.550412 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.550416 | controller | 16:08:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.550420 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:31 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.550424 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.550429 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.550433 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.550437 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.550441 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.550446 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.550450 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.550454 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.550458 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.550463 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.550467 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.550471 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.550475 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.550479 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.550484 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.550488 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.550503 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.550508 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.550512 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.550516 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.550521 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.550525 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.550533 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.550537 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.550541 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.550546 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.550550 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.550554 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.550560 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.550565 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.550570 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.550574 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.550578 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.550583 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.550587 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.550591 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.550595 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:31 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.550600 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.550604 | controller | 16:08:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.550608 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:32 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.550612 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.550616 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.550621 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.550625 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.550629 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.550633 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.550638 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.550642 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.550646 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.550650 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.550654 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.550659 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.550663 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.550667 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.550672 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.550676 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.550680 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.550704 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.550711 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.550717 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.550722 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.550726 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.550730 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.550735 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.550739 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.550743 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.550747 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.550751 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.550756 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.550760 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.550764 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.550768 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.550773 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.550777 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.550781 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.550785 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.550790 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:32 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.550794 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.550798 | controller | 16:08:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.550802 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:33 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.550807 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.550813 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.550817 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.550822 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.550826 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.550830 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.550834 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.550839 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.550843 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.550847 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.550851 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.550855 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.550861 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.550865 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.550870 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.550874 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.550878 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.550882 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.550894 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.550899 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.550903 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.550907 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.550912 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.550916 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.550920 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.550925 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.550929 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.550933 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.550937 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.550942 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.550946 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.550950 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.550954 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.550959 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.550963 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.550967 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.550971 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:33 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.550976 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.550980 | controller | 16:08:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.550984 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:34 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.550988 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.550993 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.550997 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.551005 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.551009 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.551013 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.551018 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.551022 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.551026 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.551030 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.551034 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.551039 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.551053 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.551057 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.551061 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.551065 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.551070 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.551074 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.551078 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.551093 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.551098 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.551102 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.551106 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.551111 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.551115 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.551119 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.551131 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.551135 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.551139 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.551143 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.551148 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.551152 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.551156 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.551160 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.551165 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.551169 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.551175 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:34 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.551179 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.551183 | controller | 16:08:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.551188 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:35 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.551192 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.551196 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.551200 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.551205 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.551209 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.551213 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.551243 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.551247 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.551251 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.551256 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.551260 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.551264 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.551268 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.551273 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.551277 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.551281 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.551285 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.551290 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.551294 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.551309 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.551314 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.551325 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.551330 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.551334 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.551338 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.551342 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.551347 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.551351 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.551357 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.551361 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.551366 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.551370 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.551374 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.551378 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.551383 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.551387 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.551391 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:35 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.551395 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.551399 | controller | 16:08:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.551404 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:36 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.551410 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.551414 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.551418 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.551422 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.551428 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.551433 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.551449 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.551453 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.551457 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.551461 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.551466 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.551470 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.551474 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.551478 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.551483 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.551489 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.551493 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.551497 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.551502 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.551509 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.551513 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.551518 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.551522 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.551533 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.551538 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.551542 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.551547 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.551551 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.551555 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.551559 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.551564 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.551568 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.551572 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.551576 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.551581 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.551585 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.551589 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:36 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.551593 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.551597 | controller | 16:08:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.551602 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:38 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.551606 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.551610 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.551614 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.551619 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.551623 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.551627 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.551631 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.551635 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.551640 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.551644 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.551648 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.551652 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.551658 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.551662 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.551666 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.551672 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.551677 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.551681 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.551685 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.551689 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.551710 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.551716 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.551721 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.551725 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.551735 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.551740 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.551744 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.551748 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.551753 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.551757 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.551761 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.551765 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.551770 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.551774 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.551778 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.551782 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.551787 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:38 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.551791 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.551795 | controller | 16:08:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.551799 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:39 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.551804 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.551808 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.551812 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.551818 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.551822 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.551827 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.551831 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.551835 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.551839 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.551843 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.551848 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.551852 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.551856 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.551860 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.551865 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.551869 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.551873 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.551877 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.551881 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.551886 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.551890 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.551894 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.551898 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.551902 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.551907 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.551919 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.551924 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.551928 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.551932 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.551938 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.551942 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.551947 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.551951 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.551955 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.551959 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.551964 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.551969 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:39 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.551974 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.551978 | controller | 16:08:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.551984 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:40 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.551988 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.551992 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.551996 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.552000 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.552005 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.552009 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.552013 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.552017 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.552022 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.552026 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.552030 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.552034 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.552038 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.552043 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.552047 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.552051 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.552055 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.552059 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.552063 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.552068 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.552072 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.552076 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.552080 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.552084 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.552088 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.552093 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.552097 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.552108 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.552114 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.552118 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.552123 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.552127 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.552131 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.552135 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.552140 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.552144 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.552148 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:40 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.552152 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.552165 | controller | 16:08:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.552169 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:41 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.552173 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.552177 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.552181 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.552186 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.552190 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.552194 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.552198 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.552202 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.552206 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.552211 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.552215 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.552219 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.552223 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.552229 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.552233 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.552237 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.552242 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.552246 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.552250 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.552257 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.552261 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.552266 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.552270 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.552274 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.552278 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.552282 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.552286 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.552291 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.552302 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.552306 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.552310 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.552315 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.552319 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.552323 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.552327 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.552331 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.552335 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:41 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.552340 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.552344 | controller | 16:08:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.552348 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:42 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.552352 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.552356 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.552360 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.552366 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.552370 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.552375 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.552379 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.552383 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.552387 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.552391 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.552395 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.552400 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.552405 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.552409 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.552414 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.552418 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.552422 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.552426 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.552430 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.552434 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.552439 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.552443 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.552447 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.552451 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.552455 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.552459 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.552464 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.552468 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.552474 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.552478 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.552489 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.552493 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.552497 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.552502 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.552506 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.552510 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.552514 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:42 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.552520 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.552524 | controller | 16:08:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.552528 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:43 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.552533 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.552537 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.552541 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.552548 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.552552 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.552556 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.552561 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.552565 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.552569 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.552573 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.552577 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.552582 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.552586 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.552597 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.552602 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.552606 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.552610 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.552614 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.552618 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.552622 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.552627 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.552631 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.552635 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.552639 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.552643 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.552648 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.552652 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.552656 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.552660 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.552664 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.552668 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.552679 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.552683 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.552688 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.552702 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.552709 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.552715 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:43 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.552719 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.552723 | controller | 16:08:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.552727 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:44 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.552731 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.552736 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.552740 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.552744 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.552748 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.552752 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.552756 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.552760 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.552765 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.552769 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.552773 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.552777 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.552783 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.552787 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.552793 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.552797 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.552801 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.552805 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.552809 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.552814 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.552818 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.552822 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.552826 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.552830 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.552834 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.552838 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.552842 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.552847 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.552852 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.552856 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.552861 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.552865 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.552876 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.552881 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.552885 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.552889 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.552893 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:44 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.552898 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.552902 | controller | 16:08:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.552906 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:46 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.552910 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.552914 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.552918 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.552923 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.552927 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.552931 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.552935 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.552939 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.552943 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.552947 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.552952 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.552956 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.552960 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.552964 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.552968 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.552972 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.552976 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.552981 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.552985 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.552990 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.552995 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.552999 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.553003 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.553007 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.553011 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.553015 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.553019 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.553024 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.553028 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.553032 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.553036 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.553040 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.553044 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.553063 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.553068 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.553073 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.553077 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:46 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.553081 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.553085 | controller | 16:08:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.553089 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:47 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.553093 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.553098 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.553102 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.553106 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.553110 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.553114 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.553118 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.553123 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.553127 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.553139 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.553143 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.553147 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.553153 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.553158 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.553162 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.553166 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.553170 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.553175 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.553179 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.553185 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.553189 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.553193 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.553206 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.553210 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.553214 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.553218 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.553223 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.553227 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.553231 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.553236 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.553241 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.553245 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.553249 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.553253 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.553257 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.553269 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.553274 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:47 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.553278 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.553283 | controller | 16:08:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.553287 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:48 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.553291 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.553295 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.553299 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.553305 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.553309 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.553313 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.553318 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.553322 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.553326 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.553330 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.553334 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.553340 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.553344 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.553348 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.553352 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.553356 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.553361 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.553365 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.553369 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.553373 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.553377 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.553381 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.553386 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.553391 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.553396 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.553402 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.553407 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.553413 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.553419 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.553426 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.553431 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.553437 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.553442 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.553447 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.553453 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.553458 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.553476 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:48 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.553482 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.553487 | controller | 16:08:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.553493 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:49 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.553499 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.553504 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.553508 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.553512 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.553516 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.553520 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.553525 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.553529 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.553533 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.553537 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.553541 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.553554 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.553559 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.553563 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.553567 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.553571 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.553575 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.553579 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.553584 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.553588 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.553592 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.553596 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.553600 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.553604 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.553610 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.553615 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.553621 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.553627 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.553636 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.553641 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.553647 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.553655 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.553659 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.553664 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.553668 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.553672 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.553676 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:49 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.553683 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.553709 | controller | 16:08:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.553717 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:50 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.553721 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.553725 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.553729 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.553735 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.553739 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.553744 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.553748 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.553752 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.553756 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.553760 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.553764 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.553769 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.553773 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.553777 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.553781 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.553785 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.553789 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.553794 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.553798 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.553804 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.553808 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.553812 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.553816 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.553821 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.553825 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.553829 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.553833 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.553837 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.553841 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.553845 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.553850 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.553854 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.553858 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.553862 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.553866 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.553870 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.553875 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:50 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.553879 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.553883 | controller | 16:08:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.553895 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:51 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.553899 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.553904 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.553908 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.553912 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.553916 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.553920 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.553924 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.553929 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.553933 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.553937 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.553943 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.553947 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.553953 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.553957 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.553961 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.553965 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.553969 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.553973 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.553978 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.553982 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.553986 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.553990 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.553994 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.553999 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.554003 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.554007 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.554011 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.554015 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.554019 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.554024 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.554028 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.554032 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.554038 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.554042 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.554046 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.554050 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.554055 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:51 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.554059 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.554063 | controller | 16:08:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.554067 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:53 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.554078 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.554082 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.554086 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.554092 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.554097 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.554101 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.554105 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.554109 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.554113 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.554117 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.554122 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.554126 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.554130 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.554134 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.554140 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.554144 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.554148 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.554153 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.554157 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.554162 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.554166 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.554171 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.554175 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.554179 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.554183 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.554187 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.554191 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.554197 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.554201 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.554205 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.554210 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.554214 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.554218 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.554222 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.554226 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.554230 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.554236 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:53 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.554240 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.554244 | controller | 16:08:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.554249 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:54 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.554253 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.554257 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.554268 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.554272 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.554277 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.554281 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.554285 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.554289 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.554293 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.554298 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.554302 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.554306 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.554310 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.554314 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.554318 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.554323 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.554327 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.554331 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.554335 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.554339 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.554343 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.554347 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.554351 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.554356 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.554360 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.554364 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.554368 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.554372 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.554378 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.554382 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.554386 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.554390 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.554394 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.554398 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.554403 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.554407 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.554411 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:54 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.554415 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.554419 | controller | 16:08:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.554424 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:55 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.554428 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.554432 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.554436 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.554446 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.554451 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.554455 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.554459 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.554464 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.554468 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.554473 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.554478 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.554482 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.554486 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.554490 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.554494 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.554498 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.554503 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.554507 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.554511 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.554517 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.554521 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.554525 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.554529 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.554533 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.554537 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.554542 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.554546 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.554550 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.554554 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.554560 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.554564 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.554568 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.554573 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.554577 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.554581 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.554585 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.554589 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:55 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.554593 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.554597 | controller | 16:08:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.554602 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:56 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.554606 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.554610 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.554614 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.554618 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.554629 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.554633 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.554637 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.554641 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.554646 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.554650 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.554654 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.554658 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.554665 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.554669 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.554673 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.554678 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.554682 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.554686 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.554690 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.554706 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.554711 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.554715 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.554719 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.554723 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.554727 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.554731 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.554737 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.554742 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.554746 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.554750 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.554754 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.554758 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.554762 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.554767 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.554771 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.554775 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.554779 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:56 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.554783 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.554787 | controller | 16:08:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.554791 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:57 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.554796 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.554800 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.554804 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.554809 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.554814 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.554818 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.554829 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.554834 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.554838 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.554842 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.554846 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.554851 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.554855 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.554859 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.554863 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.554867 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.554871 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.554875 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.554880 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.554884 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.554888 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.554892 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.554896 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.554901 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.554905 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.554909 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.554913 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.554917 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.554921 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.554926 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.554930 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.554934 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.554938 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.554942 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.554946 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.554951 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.554956 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:57 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.554961 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.554965 | controller | 16:08:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.554969 | controller | \"ac-watcher\" not found\n logger.go:42: 16:08:58 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.554973 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.554977 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.554981 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.554987 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.554991 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.554995 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.555000 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.555010 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.555016 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.555020 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.555024 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.555029 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.555033 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.555037 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.555041 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.555045 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.555049 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.555053 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.555058 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.555063 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.555067 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.555071 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.555076 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.555089 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.555093 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.555098 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.555102 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.555106 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.555112 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.555116 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.555120 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.555124 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.555128 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.555133 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.555137 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.555141 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.555145 | controller | expected application_credential_id\"]\n logger.go:42: 16:08:58 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.555149 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.555153 | controller | 16:08:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.555158 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:00 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.555169 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.555173 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.555177 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.555182 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.555186 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.555190 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.555194 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.555198 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.555202 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.555213 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.555217 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.555221 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.555225 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.555229 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.555234 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.555238 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.555250 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.555254 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.555258 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.555264 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.555268 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.555272 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.555276 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.555280 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.555286 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.555291 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.555295 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.555299 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.555303 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.555307 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.555311 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.555315 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.555320 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.555324 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.555328 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.555332 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.555336 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:00 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.555340 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.555344 | controller | 16:09:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.555349 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:01 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.555353 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.555357 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.555361 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.555365 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.555369 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.555373 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.555378 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.555382 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.555386 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.555390 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.555400 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.555405 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.555410 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.555415 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.555420 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.555424 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.555429 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.555433 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.555437 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.555457 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.555461 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.555465 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.555470 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.555474 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.555478 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.555482 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.555486 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.555490 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.555494 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.555499 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.555503 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.555507 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.555511 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.555515 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.555519 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.555524 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.555528 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:01 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.555532 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.555536 | controller | 16:09:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.555540 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:02 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.555545 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.555549 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.555553 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.555559 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.555563 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.555567 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.555571 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.555577 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.555581 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.555585 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.555589 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.555594 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.555603 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.555608 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.555612 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.555616 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.555620 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.555624 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.555629 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.555633 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.555637 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.555641 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.555645 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.555649 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.555653 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.555657 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.555661 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.555666 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.555670 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.555674 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.555678 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.555682 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.555686 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.555690 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.555714 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.555719 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.555728 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:02 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.555732 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.555737 | controller | 16:09:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.555741 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:03 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.555745 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.555749 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.555753 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.555757 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.555763 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.555767 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.555772 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.555784 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.555789 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.555793 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.555797 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.555801 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.555805 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.555816 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.555820 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.555825 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.555829 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.555833 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.555837 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.555841 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.555845 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.555850 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.555854 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.555858 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.555862 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.555866 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.555872 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.555876 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.555882 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.555887 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.555891 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.555897 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.555901 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.555905 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.555909 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.555914 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.555918 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:03 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.555922 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.555926 | controller | 16:09:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.555930 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:04 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.555934 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.555938 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.555943 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.555947 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.555951 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.555955 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.555959 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.555963 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.555968 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.555972 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.555976 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.555980 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.555984 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.555989 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.555998 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.556003 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.556007 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.556011 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.556015 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.556023 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.556027 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.556031 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.556035 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.556039 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.556043 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.556048 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.556052 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.556056 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.556060 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.556064 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.556068 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.556072 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.556077 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.556081 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.556085 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.556089 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.556093 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:04 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.556097 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.556102 | controller | 16:09:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.556106 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:05 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.556110 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.556114 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.556120 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.556124 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.556128 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.556132 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.556136 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.556140 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.556144 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.556149 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.556153 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.556157 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.556163 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.556167 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.556171 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.556175 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.556185 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.556190 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.556194 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.556198 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.556202 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.556207 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.556211 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.556215 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.556219 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.556223 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.556227 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.556231 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.556236 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.556240 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.556244 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.556248 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.556252 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.556256 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.556261 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.556265 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.556269 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:05 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.556273 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.556277 | controller | 16:09:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.556281 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:07 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.556286 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.556290 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.556294 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.556301 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.556305 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.556309 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.556313 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.556318 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.556322 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.556326 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.556330 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.556334 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.556338 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.556343 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.556347 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.556351 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.556355 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.556365 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.556370 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.556374 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.556378 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.556382 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.556388 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.556392 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.556396 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.556400 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.556404 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.556409 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.556413 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.556417 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.556421 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.556425 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.556430 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.556434 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.556438 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.556442 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.556448 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:07 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.556452 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.556456 | controller | 16:09:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.556460 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:08 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.556464 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.556469 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.556473 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.556477 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.556481 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.556485 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.556489 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.556493 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.556497 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.556502 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.556506 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.556510 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.556514 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.556518 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.556522 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.556527 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.556531 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.556535 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.556545 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.556549 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.556553 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.556557 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.556561 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.556566 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.556570 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.556574 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.556578 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.556582 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.556588 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.556592 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.556596 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.556600 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.556605 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.556609 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.556613 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.556617 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.556621 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:08 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.556625 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.556629 | controller | 16:09:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.556634 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:09 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.556638 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.556644 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.556648 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.556652 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.556656 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.556660 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.556664 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.556669 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.556673 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.556677 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.556681 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.556685 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.556689 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.556707 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.556714 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.556719 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.556723 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.556727 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.556731 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.556743 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.556748 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.556752 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.556756 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.556760 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.556765 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.556769 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.556773 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.556777 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.556781 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.556785 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.556789 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.556793 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.556798 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.556802 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.556806 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.556810 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.556814 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:09 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.556818 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.556822 | controller | 16:09:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.556835 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:10 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.556839 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.556843 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.556847 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.556851 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.556856 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.556860 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.556864 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.556868 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.556872 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.556876 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.556881 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.556885 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.556890 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.556901 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.556905 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.556910 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.556914 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.556918 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.556922 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.556928 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.556942 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.556953 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.556957 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.556961 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.556966 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.556970 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.556974 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.556978 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.556982 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.556993 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.556998 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.557002 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.557006 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.557010 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.557014 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.557018 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.557023 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:10 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.557027 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.557031 | controller | 16:09:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.557035 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:11 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.557039 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.557044 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.557048 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.557054 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.557058 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.557062 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.557067 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.557071 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.557075 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.557087 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.557091 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.557095 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.557100 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.557104 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.557108 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.557112 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.557116 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.557120 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.557125 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.557129 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.557133 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.557137 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.557147 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.557151 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.557156 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.557160 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.557164 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.557168 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.557172 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.557178 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.557182 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.557187 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.557191 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.557195 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.557199 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.557203 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.557209 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:11 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.557213 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.557217 | controller | 16:09:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.557221 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:13 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.557227 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.557231 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.557235 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.557239 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.557244 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.557248 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.557252 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.557256 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.557260 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.557264 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.557269 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.557273 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.557277 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.557281 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.557285 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.557289 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.557293 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.557298 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.557302 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.557306 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.557310 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.557314 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.557318 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.557322 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.557332 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.557337 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.557341 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.557345 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.557351 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.557355 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.557359 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.557363 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.557368 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.557372 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.557376 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.557380 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.557384 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:13 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.557388 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.557392 | controller | 16:09:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.557397 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:14 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.557401 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.557405 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.557409 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.557413 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.557417 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.557421 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.557426 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.557430 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.557434 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.557438 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.557442 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.557446 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.557451 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.557455 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.557459 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.557465 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.557469 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.557473 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.557477 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.557483 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.557487 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.557491 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.557495 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.557499 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.557503 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.557514 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.557518 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.557522 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.557526 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.557530 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.557535 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.557539 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.557543 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.557547 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.557558 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.557562 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.557566 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:14 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.557571 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.557575 | controller | 16:09:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.557579 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:15 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.557583 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.557587 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.557591 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.557597 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.557601 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.557606 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.557610 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.557614 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.557618 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.557622 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.557626 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.557631 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.557636 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.557641 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.557645 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.557649 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.557653 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.557657 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.557661 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.557666 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.557670 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.557674 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.557678 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.557682 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.557686 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.557691 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.557707 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.557718 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.557723 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.557727 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.557731 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.557735 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.557740 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.557744 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.557748 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.557752 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.557756 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:15 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.557769 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.557773 | controller | 16:09:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.557779 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:16 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.557783 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.557788 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.557792 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.557799 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.557803 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.557807 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.557811 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.557816 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.557820 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.557824 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.557828 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.557832 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.557836 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.557840 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.557845 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.557849 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.557853 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.557857 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.557861 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.557867 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.557871 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.557875 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.557879 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.557884 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.557888 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.557892 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.557896 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.557900 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.557910 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.557915 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.557919 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.557923 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.557927 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.557932 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.557936 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.557940 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.557946 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:16 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.557952 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.557956 | controller | 16:09:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.557960 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:17 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.557964 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.557969 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.557973 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.557977 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.557981 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.557985 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.557989 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.557994 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.557998 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.558002 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.558006 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.558010 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.558014 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.558020 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.558025 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.558030 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.558034 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.558038 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.558042 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.558046 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.558050 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.558054 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.558059 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.558063 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.558067 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.558071 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.558075 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.558079 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.558085 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.558103 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.558108 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.558112 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.558116 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.558120 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.558124 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.558128 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.558133 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:17 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.558137 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.558141 | controller | 16:09:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.558145 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:18 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.558149 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.558153 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.558158 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.558162 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.558166 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.558170 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.558174 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.558178 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.558182 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.558187 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.558191 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.558195 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.558199 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.558203 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.558207 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.558211 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.558216 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.558220 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.558224 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.558230 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.558234 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.558238 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.558242 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.558246 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.558250 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.558255 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.558259 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.558263 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.558267 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.558271 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.558275 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.558285 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.558290 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.558294 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.558298 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.558302 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.558306 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:18 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.558312 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.558317 | controller | 16:09:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.558321 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:20 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.558325 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.558329 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.558333 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.558337 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.558342 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.558346 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.558350 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.558354 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.558358 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.558362 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.558366 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.558370 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.558376 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.558380 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.558385 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.558389 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.558393 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.558397 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.558401 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.558405 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.558410 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.558414 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.558418 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.558422 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.558426 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.558430 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.558434 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.558439 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.558443 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.558448 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.558453 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.558457 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.558468 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.558472 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.558477 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.558481 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.558485 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:20 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.558489 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.558493 | controller | 16:09:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.558498 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:21 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.558502 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.558506 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.558510 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.558516 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.558520 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.558524 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.558528 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.558532 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.558536 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.558541 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.558545 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.558549 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.558555 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.558559 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.558563 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.558567 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.558571 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.558576 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.558580 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.558584 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.558588 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.558592 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.558596 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.558601 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.558605 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.558609 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.558613 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.558617 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.558621 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.558625 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.558630 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.558634 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.558638 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.558648 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.558652 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.558656 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.558662 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:21 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.558666 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.558678 | controller | 16:09:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.558682 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:22 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.558687 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.558691 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.558707 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.558711 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.558716 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.558720 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.558724 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.558728 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.558732 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.558736 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.558740 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.558745 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.558756 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.558760 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.558764 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.558768 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.558773 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.558777 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.558781 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.558787 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.558791 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.558795 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.558799 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.558804 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.558808 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.558812 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.558816 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.558820 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.558835 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.558839 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.558844 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.558848 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.558852 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.558856 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.558862 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.558874 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.558879 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:22 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.558883 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.558887 | controller | 16:09:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.558892 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:23 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.558896 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.558900 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.558904 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.558910 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.558914 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.558918 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.558922 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.558927 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.558931 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.558935 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.558939 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.558943 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.558947 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.558951 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.558956 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.558960 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.558964 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.558968 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.558972 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.558978 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.558982 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.558986 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.558991 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.558995 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.558999 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.559003 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.559007 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.559011 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.559016 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.559020 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.559024 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.559028 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.559032 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.559036 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.559041 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.559045 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.559056 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:23 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.559061 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.559065 | controller | 16:09:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.559069 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:24 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.559073 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.559077 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.559082 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.559086 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.559090 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.559094 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.559098 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.559102 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.559106 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.559111 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.559115 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.559121 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.559126 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.559131 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.559135 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.559139 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.559143 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.559147 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.559152 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.559156 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.559160 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.559164 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.559168 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.559172 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.559177 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.559181 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.559185 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.559196 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.559200 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.559205 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.559209 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.559213 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.559217 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.559221 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.559225 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.559230 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.559234 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:24 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.559244 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.559249 | controller | 16:09:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.559253 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:25 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.559257 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.559261 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.559266 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.559271 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.559275 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.559280 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.559284 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.559288 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.559292 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.559296 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.559300 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.559304 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.559309 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.559313 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.559319 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.559323 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.559327 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.559331 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.559335 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.559340 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.559344 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.559348 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.559352 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.559356 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.559360 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.559364 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.559369 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.559373 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.559377 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.559381 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.559385 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.559391 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.559395 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.559399 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.559403 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.559407 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.559413 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:25 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.559417 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.559421 | controller | 16:09:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.559434 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:27 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.559439 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.559457 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.559461 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.559465 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.559470 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.559474 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.559478 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.559482 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.559486 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.559490 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.559494 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.559499 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.559503 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.559507 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.559511 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.559515 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.559519 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.559523 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.559528 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.559532 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.559536 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.559540 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.559544 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.559548 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.559552 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.559557 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.559561 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.559565 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.559571 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.559575 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.559579 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.559583 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.559588 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.559592 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.559596 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.559600 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.559604 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:27 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.559608 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.559613 | controller | 16:09:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.559617 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:28 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.559628 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.559633 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.559637 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.559641 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.559645 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.559649 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.559654 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.559658 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.559662 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.559666 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.559672 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.559676 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.559681 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.559685 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.559689 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.559703 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.559710 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.559714 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.559718 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.559728 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.559732 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.559736 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.559741 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.559745 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.559749 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.559753 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.559757 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.559761 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.559765 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.559771 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.559775 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.559780 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.559787 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.559791 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.559795 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.559799 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.559803 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:28 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.559807 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.559812 | controller | 16:09:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.559816 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:29 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.559820 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.559831 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.559836 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.559840 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.559844 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.559849 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.559853 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.559857 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.559861 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.559865 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.559869 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.559873 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.559879 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.559883 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.559888 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.559892 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.559896 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.559900 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.559904 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.559909 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.559913 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.559917 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.559921 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.559925 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.559930 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.559934 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.559938 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.559944 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.559948 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.559952 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.559956 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.559960 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.559964 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.559969 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.559973 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.559977 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.559981 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:29 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.559985 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.559990 | controller | 16:09:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.559994 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:30 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.559998 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.560002 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.560006 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.560019 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.560025 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.560029 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.560034 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.560038 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.560042 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.560046 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.560050 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.560055 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.560059 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.560063 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.560067 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.560071 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.560076 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.560080 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.560084 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.560088 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.560092 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.560097 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.560101 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.560105 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.560109 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.560113 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.560117 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.560122 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.560126 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.560130 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.560134 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.560138 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.560143 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.560147 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.560151 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.560155 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.560161 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:30 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.560165 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.560169 | controller | 16:09:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.560174 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:31 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.560178 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.560182 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.560186 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.560192 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.560202 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.560207 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.560211 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.560215 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.560219 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.560225 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.560229 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.560233 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.560238 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.560242 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.560246 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.560250 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.560254 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.560258 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.560262 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.560267 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.560271 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.560275 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.560279 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.560283 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.560287 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.560291 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.560296 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.560300 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.560305 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.560310 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.560314 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.560318 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.560322 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.560326 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.560331 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.560335 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.560339 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:31 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.560343 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.560347 | controller | 16:09:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.560351 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:32 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.560355 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.560360 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.560364 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.560368 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.560372 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.560376 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.560386 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.560390 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.560395 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.560399 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.560403 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.560407 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.560411 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.560416 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.560420 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.560424 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.560428 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.560432 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.560436 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.560442 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.560446 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.560451 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.560455 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.560459 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.560463 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.560467 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.560473 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.560477 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.560481 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.560485 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.560490 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.560494 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.560498 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.560502 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.560506 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.560510 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.560514 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:32 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.560518 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.560523 | controller | 16:09:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.560527 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:34 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.560531 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.560535 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.560539 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.560543 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.560548 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.560552 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.560556 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.560574 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.560579 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.560583 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.560587 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.560591 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.560597 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.560601 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.560607 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.560611 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.560615 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.560620 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.560624 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.560629 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.560634 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.560638 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.560642 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.560654 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.560658 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.560662 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.560667 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.560671 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.560675 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.560679 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.560683 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.560687 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.560702 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.560708 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.560712 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.560717 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.560729 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:34 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.560733 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.560737 | controller | 16:09:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.560741 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:35 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.560746 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.560750 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.560754 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.560760 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.560764 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.560768 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.560772 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.560777 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.560790 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.560795 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.560799 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.560804 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.560808 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.560812 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.560816 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.560820 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.560824 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.560829 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.560833 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.560837 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.560841 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.560845 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.560849 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.560854 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.560858 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.560862 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.560866 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.560870 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.560874 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.560879 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.560883 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.560887 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.560891 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.560896 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.560900 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.560904 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.560910 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:35 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.560914 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.560918 | controller | 16:09:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.560922 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:36 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.560927 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.560931 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.560935 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.560939 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.560943 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.560948 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.560952 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.560956 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.560960 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.560964 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.560975 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.560979 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.560984 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.560988 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.560992 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.560996 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.561000 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.561004 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.561008 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.561013 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.561017 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.561021 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.561025 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.561030 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.561035 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.561040 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.561044 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.561048 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.561054 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.561059 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.561063 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.561068 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.561072 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.561076 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.561080 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.561084 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.561089 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:36 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.561093 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.561097 | controller | 16:09:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.561101 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:37 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.561105 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.561109 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.561114 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.561118 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.561122 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.561126 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.561130 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.561134 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.561139 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.561143 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.561147 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.561157 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.561162 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.561166 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.561178 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.561182 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.561186 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.561191 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.561195 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.561200 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.561205 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.561209 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.561213 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.561217 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.561221 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.561226 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.561230 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.561234 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.561238 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.561242 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.561246 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.561251 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.561255 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.561259 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.561263 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.561267 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.561272 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:37 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.561276 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.561280 | controller | 16:09:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.561284 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:38 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.561289 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.561293 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.561297 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.561301 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.561305 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.561309 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.561314 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.561319 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.561324 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.561328 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.561332 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.561336 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.561342 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.561353 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.561357 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.561361 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.561366 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.561370 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.561374 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.561378 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.561382 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.561386 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.561391 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.561395 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.561399 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.561403 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.561407 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.561411 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.561415 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.561419 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.561424 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.561428 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.561432 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.561436 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.561440 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.561444 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.561449 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:38 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.561453 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.561457 | controller | 16:09:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.561461 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:39 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.561465 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.561470 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.561474 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.561481 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.561485 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.561489 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.561494 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.561498 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.561502 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.561506 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.561510 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.561514 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.561519 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.561523 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.561534 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.561538 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.561542 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.561546 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.561551 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.561556 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.561561 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.561565 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.561569 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.561574 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.561579 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.561583 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.561587 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.561591 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.561595 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.561599 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.561604 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.561608 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.561612 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.561616 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.561620 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.561624 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.561630 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:39 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.561634 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.561639 | controller | 16:09:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.561643 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:41 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.561647 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.561651 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.561655 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.561659 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.561664 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.561668 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.561672 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.561676 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.561680 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.561684 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.561689 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.561704 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.561711 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.561715 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.561719 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.561723 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.561734 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.561739 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.561743 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.561747 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.561752 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.561756 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.561760 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.561764 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.561768 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.561772 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.561776 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.561781 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.561787 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.561791 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.561795 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.561799 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.561803 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.561808 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.561812 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.561816 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.561820 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:41 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.561824 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.561828 | controller | 16:09:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.561833 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:42 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.561837 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.561841 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.561847 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.561851 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.561855 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.561859 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.561863 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.561868 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.561872 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.561876 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.561880 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.561884 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.561888 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.561893 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.561900 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.561904 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.561908 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.561919 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.561924 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.561930 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.561934 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.561938 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.561942 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.561947 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.561951 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.561955 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.561959 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.561963 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.561967 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.561971 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.561976 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.561980 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.561984 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.561988 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.561992 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.561996 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.562001 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:42 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.562005 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.562009 | controller | 16:09:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.562013 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:43 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.562017 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.562021 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.562026 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.562030 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.562034 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.562038 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.562042 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.562047 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.562051 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.562055 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.562059 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.562063 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.562069 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.562073 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.562077 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.562081 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.562087 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.562093 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.562105 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.562111 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.562117 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.562122 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.562130 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.562136 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.562142 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.562147 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.562153 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.562159 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.562164 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.562181 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.562186 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.562192 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.562199 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.562203 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.562207 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.562212 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.562216 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:43 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.562221 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.562226 | controller | 16:09:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.562230 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:44 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.562234 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.562239 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.562244 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.562263 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.562269 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.562274 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.562280 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.562285 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.562291 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.562297 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.562302 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.562308 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.562313 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.562319 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.562323 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.562340 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.562344 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.562349 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.562353 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.562367 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.562372 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.562379 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.562385 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.562389 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.562393 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.562398 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.562402 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.562406 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.562410 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.562418 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.562423 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.562427 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.562431 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.562435 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.562440 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.562444 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.562450 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:44 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.562454 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.562458 | controller | 16:09:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.562463 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:45 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.562467 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.562473 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.562477 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.562481 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.562485 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.562489 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.562494 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.562498 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.562502 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.562506 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.562510 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.562514 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.562519 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.562523 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.562527 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.562531 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.562535 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.562540 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.562544 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.562549 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.562554 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.562566 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.562570 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.562574 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.562579 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.562583 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.562587 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.562591 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.562597 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.562602 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.562606 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.562610 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.562614 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.562618 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.562622 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.562627 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.562631 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:45 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.562635 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.562639 | controller | 16:09:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.562643 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:46 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.562656 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.562660 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.562664 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.562668 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.562672 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.562676 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.562681 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.562685 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.562689 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.562705 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.562712 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.562716 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.562720 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.562724 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.562728 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.562733 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.562737 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.562741 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.562745 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.562751 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.562759 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.562764 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.562779 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.562785 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.562789 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.562794 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.562799 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.562805 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.562810 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.562816 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.562821 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.562825 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.562830 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.562834 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.562838 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.562842 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.562847 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:46 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.562851 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.562857 | controller | 16:09:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.562863 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:47 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.562869 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.562875 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.562881 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.562891 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.562897 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.562902 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.562906 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.562910 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.562915 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.562919 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.562923 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.562927 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.562934 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.562938 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.562942 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.562947 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.562951 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.562955 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.562959 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.562964 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.562968 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.562972 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.562976 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.562981 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.562995 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.563000 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.563004 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.563008 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.563012 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.563017 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.563021 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.563025 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.563029 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.563034 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.563038 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.563042 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.563047 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:47 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.563053 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.563059 | controller | 16:09:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.563065 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:49 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.563073 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.563079 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.563084 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.563091 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.563095 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.563099 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.563103 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.563107 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.563112 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.563116 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.563120 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.563124 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.563128 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.563132 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.563136 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.563140 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.563145 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.563149 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.563153 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.563157 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.563161 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.563165 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.563170 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.563174 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.563178 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.563191 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.563195 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.563200 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.563204 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.563208 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.563212 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.563216 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.563220 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.563225 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.563229 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.563233 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.563239 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:49 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.563244 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.563250 | controller | 16:09:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.563255 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:50 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.563261 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.563266 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.563270 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.563274 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.563278 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.563282 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.563286 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.563291 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.563295 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.563299 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.563303 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.563307 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.563311 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.563316 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.563322 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.563327 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.563332 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.563336 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.563340 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.563344 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.563348 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.563352 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.563356 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.563361 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.563365 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.563369 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.563373 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.563386 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.563392 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.563396 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.563401 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.563405 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.563409 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.563413 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.563417 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.563421 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.563426 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:50 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.563430 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.563434 | controller | 16:09:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.563438 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:51 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.563442 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.563446 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.563475 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.563481 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.563485 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.563489 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.563493 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.563497 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.563502 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.563506 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.563510 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.563514 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.563518 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.563522 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.563526 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.563530 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.563535 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.563539 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.563543 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.563551 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.563555 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.563559 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.563564 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.563568 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.563572 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.563576 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.563580 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.563584 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.563596 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.563601 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.563605 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.563609 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.563614 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.563618 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.563622 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.563626 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.563630 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:51 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.563634 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.563639 | controller | 16:09:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.563645 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:52 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.563649 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.563653 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.563657 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.563661 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.563666 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.563670 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.563674 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.563678 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.563682 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.563686 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.563690 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.563710 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.563716 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.563720 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.563725 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.563729 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.563733 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.563737 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.563741 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.563745 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.563750 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.563754 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.563758 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.563762 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.563766 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.563770 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.563774 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.563779 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.563783 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.563796 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.563801 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.563805 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.563809 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.563813 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.563817 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.563822 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.563826 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:52 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.563830 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.563834 | controller | 16:09:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.563838 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:53 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.563842 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.563846 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.563851 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.563856 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.563861 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.563865 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.563880 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.563884 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.563888 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.563892 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.563897 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.563901 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.563905 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.563911 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.563915 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.563919 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.563923 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.563927 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.563940 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.563944 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.563948 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.563953 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.563957 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.563961 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.563965 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.563969 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.563973 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.563978 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.563982 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.563986 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.563990 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.564007 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.564012 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.564016 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.564021 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.564025 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.564032 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:53 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.564036 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.564041 | controller | 16:09:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.564045 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:54 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.564049 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.564053 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.564057 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.564061 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.564065 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.564070 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.564074 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.564078 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.564082 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.564086 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.564091 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.564103 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.564107 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.564111 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.564116 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.564120 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.564124 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.564128 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.564132 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.564136 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.564141 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.564145 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.564149 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.564153 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.564157 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.564161 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.564165 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.564170 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.564175 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.564180 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.564184 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.564188 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.564199 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.564203 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.564208 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.564212 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.564216 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:54 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.564222 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.564226 | controller | 16:09:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.564231 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:56 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.564235 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.564239 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.564243 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.564249 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.564253 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.564257 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.564261 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.564266 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.564270 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.564274 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.564278 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.564282 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.564286 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.564291 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.564295 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.564299 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.564303 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.564307 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.564311 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.564317 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.564321 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.564326 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.564330 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.564334 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.564338 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.564342 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.564346 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.564351 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.564355 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.564359 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.564363 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.564367 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.564371 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.564382 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.564386 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.564391 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.564395 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:56 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.564399 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.564403 | controller | 16:09:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.564407 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:57 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.564411 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.564416 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.564420 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.564424 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.564430 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.564434 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.564438 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.564443 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.564447 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.564451 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.564455 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.564459 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.564465 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.564469 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.564474 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.564479 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.564484 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.564488 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.564492 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.564496 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.564502 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.564506 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.564510 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.564514 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.564518 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.564523 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.564527 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.564531 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.564535 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.564539 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.564544 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.564548 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.564560 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.564564 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.564569 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.564579 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.564583 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:57 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.564588 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.564592 | controller | 16:09:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.564596 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:58 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.564600 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.564605 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.564609 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.564615 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.564619 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.564623 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.564627 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.564631 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.564636 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.564640 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.564644 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.564648 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.564652 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.564656 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.564662 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.564667 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.564671 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.564675 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.564679 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.564683 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.564687 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.564708 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.564715 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.564720 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.564724 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.564728 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.564732 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.564736 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.564741 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.564745 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.564749 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.564753 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.564757 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.564762 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.564768 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.564772 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.564784 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:58 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.564789 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.564793 | controller | 16:09:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.564798 | controller | \"ac-watcher\" not found\n logger.go:42: 16:09:59 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.564802 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.564806 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.564810 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.564814 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.564818 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.564823 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.564827 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.564831 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.564835 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.564839 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.564844 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.564848 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.564852 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.564856 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.564860 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.564864 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.564868 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.564873 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.564877 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.564881 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.564885 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.564889 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.564893 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.564898 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.564902 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.564906 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.564910 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.564914 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.564920 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.564924 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.564929 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.564933 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.564937 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.564941 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.564945 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.564957 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.564961 | controller | expected application_credential_id\"]\n logger.go:42: 16:09:59 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.564973 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.564977 | controller | 16:09:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.564982 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:00 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.564986 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.564990 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.564994 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.564999 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.565003 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.565007 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.565011 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.565015 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.565019 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.565024 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.565028 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.565034 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.565038 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.565042 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.565046 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.565050 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.565057 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.565061 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.565066 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.565071 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.565075 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.565079 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.565084 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.565088 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.565092 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.565096 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.565100 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.565104 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.565108 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.565114 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.565118 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.565122 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.565127 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.565131 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.565135 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.565139 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.565143 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:00 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.565147 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.565159 | controller | 16:10:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.565163 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:01 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.565168 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.565172 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.565176 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.565180 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.565184 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.565188 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.565193 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.565197 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.565201 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.565205 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.565209 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.565213 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.565219 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.565223 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.565227 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.565232 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.565236 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.565240 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.565244 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.565248 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.565252 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.565257 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.565261 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.565265 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.565269 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.565273 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.565278 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.565282 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.565286 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.565290 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.565294 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.565300 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.565305 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.565309 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.565313 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.565317 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.565321 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:01 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.565325 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.565329 | controller | 16:10:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.565334 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:03 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.565344 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.565348 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.565352 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.565358 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.565362 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.565367 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.565371 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.565375 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.565379 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.565383 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.565387 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.565392 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.565396 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.565400 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.565404 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.565408 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.565412 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.565416 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.565421 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.565427 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.565431 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.565435 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.565439 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.565443 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.565448 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.565452 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.565456 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.565460 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.565464 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.565468 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.565472 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.565477 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.565481 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.565485 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.565489 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.565493 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.565499 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:03 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.565503 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.565507 | controller | 16:10:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.565512 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:04 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.565516 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.565526 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.565530 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.565536 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.565540 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.565544 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.565548 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.565553 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.565557 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.565561 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.565572 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.565576 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.565581 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.565585 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.565589 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.565593 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.565597 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.565602 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.565608 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.565613 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.565619 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.565625 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.565631 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.565637 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.565643 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.565648 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.565654 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.565670 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.565677 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.565681 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.565685 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.565690 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.565708 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.565713 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.565717 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.565721 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.565725 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:04 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.565737 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.565742 | controller | 16:10:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.565746 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:05 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.565750 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.565754 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.565758 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.565769 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.565774 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.565778 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.565782 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.565787 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.565791 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.565802 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.565807 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.565811 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.565815 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.565819 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.565823 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.565828 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.565832 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.565836 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.565840 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.565847 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.565852 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.565858 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.565863 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.565869 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.565874 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.565879 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.565883 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.565890 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.565894 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.565898 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.565903 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.565907 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.565911 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.565915 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.565919 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.565923 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.565928 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:05 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.565932 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.565936 | controller | 16:10:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.565940 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:06 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.565944 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.565948 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.565953 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.565957 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.565968 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.565972 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.565977 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.565981 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.565985 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.565989 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.565993 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.565998 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.566004 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.566008 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.566014 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.566018 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.566022 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.566026 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.566030 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.566035 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.566039 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.566043 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.566047 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.566051 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.566055 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.566060 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.566064 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.566068 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.566072 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.566076 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.566080 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.566084 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.566089 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.566093 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.566097 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.566101 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.566105 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:06 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.566110 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.566114 | controller | 16:10:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.566118 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:07 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.566122 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.566126 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.566130 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.566137 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.566142 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.566152 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.566156 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.566161 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.566165 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.566171 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.566175 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.566179 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.566183 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.566187 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.566191 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.566196 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.566200 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.566204 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.566208 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.566212 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.566216 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.566220 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.566225 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.566229 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.566241 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.566246 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.566250 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.566254 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.566258 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.566262 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.566266 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.566270 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.566275 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.566279 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.566283 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.566287 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.566293 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:07 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.566297 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.566301 | controller | 16:10:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.566305 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:08 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.566310 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.566314 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.566318 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.566322 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.566326 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.566330 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.566335 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.566345 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.566349 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.566353 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.566358 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.566362 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.566366 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.566370 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.566374 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.566379 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.566383 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.566387 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.566391 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.566397 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.566401 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.566405 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.566409 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.566413 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.566418 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.566422 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.566427 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.566431 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.566437 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.566443 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.566447 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.566451 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.566455 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.566459 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.566463 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.566468 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.566472 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:08 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.566476 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.566480 | controller | 16:10:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.566484 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:09 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.566488 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.566493 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.566497 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.566501 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.566505 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.566509 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.566514 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.566518 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.566528 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.566532 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.566536 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.566541 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.566545 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.566549 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.566553 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.566557 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.566561 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.566566 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.566570 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.566575 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.566580 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.566584 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.566588 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.566592 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.566596 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.566600 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.566605 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.566609 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.566613 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.566617 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.566621 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.566625 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.566630 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.566634 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.566638 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.566642 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.566646 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:09 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.566652 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.566656 | controller | 16:10:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.566661 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:11 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.566665 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.566669 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.566673 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.566677 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.566681 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.566686 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.566690 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.566707 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.566713 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.566717 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.566728 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.566732 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.566739 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.566743 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.566747 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.566751 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.566756 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.566760 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.566764 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.566768 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.566772 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.566776 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.566780 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.566784 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.566789 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.566793 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.566797 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.566801 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.566805 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.566809 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.566813 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.566818 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.566822 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.566826 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.566830 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.566834 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.566839 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:11 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.566843 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.566847 | controller | 16:10:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.566851 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:12 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.566855 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.566859 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.566863 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.566871 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.566875 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.566879 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.566883 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.566887 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.566892 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.566896 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.566900 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.566910 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.566914 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.566918 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.566923 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.566927 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.566931 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.566935 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.566939 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.566943 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.567052 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.567056 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.567060 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.567065 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.567071 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.567075 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.567080 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.567084 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.567088 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.567092 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.567096 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.567121 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.567126 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.567130 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.567135 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.567139 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.567145 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:12 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.567149 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.567153 | controller | 16:10:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.567157 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:13 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.567162 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.567166 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.567170 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.567174 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.567178 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.567183 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.567187 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.567191 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.567195 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.567199 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.567203 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.567208 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.567212 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.567223 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.567227 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.567232 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.567236 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.567240 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.567244 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.567249 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.567253 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.567257 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.567261 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.567273 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.567278 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.567282 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.567286 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.567290 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.567297 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.567301 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.567306 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.567310 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.567314 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.567318 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.567322 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.567326 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.567331 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:13 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.567342 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.567346 | controller | 16:10:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.567350 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:14 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.567355 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.567359 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.567363 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.567367 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.567371 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.567375 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.567379 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.567385 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.567390 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.567394 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.567398 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.567402 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.567413 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.567418 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.567429 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.567434 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.567438 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.567442 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.567447 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.567465 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.567470 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.567474 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.567478 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.567483 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.567488 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.567494 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.567500 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.567505 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.567511 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.567516 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.567520 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.567524 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.567528 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.567533 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.567537 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.567541 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.567545 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:14 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.567549 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.567553 | controller | 16:10:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.567558 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:15 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.567562 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.567566 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.567570 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.567574 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.567578 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.567583 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.567587 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.567591 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.567595 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.567599 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.567605 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.567610 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.567618 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.567624 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.567630 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.567635 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.567651 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.567656 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.567661 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.567665 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.567669 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.567673 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.567677 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.567684 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.567688 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.567707 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.567715 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.567721 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.567726 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.567732 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.567738 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.567743 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.567749 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.567755 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.567760 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.567766 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.567771 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:15 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.567777 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.567783 | controller | 16:10:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.567788 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:16 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.567792 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.567796 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.567800 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.567807 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.567811 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.567815 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.567820 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.567824 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.567828 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.567832 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.567836 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.567840 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.567844 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.567849 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.567853 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.567857 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.567861 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.567874 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.567879 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.567883 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.567887 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.567891 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.567896 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.567900 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.567904 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.567908 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.567912 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.567918 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.567923 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.567931 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.567937 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.567942 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.567946 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.567951 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.567955 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.567959 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.567965 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:16 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.567971 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.567976 | controller | 16:10:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.567982 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:18 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.567988 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.567992 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.567998 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.568002 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.568007 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.568011 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.568015 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.568019 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.568023 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.568028 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.568042 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.568046 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.568051 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.568057 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.568062 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.568068 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.568074 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.568079 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.568094 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.568099 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.568104 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.568108 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.568112 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.568117 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.568123 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.568128 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.568134 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.568139 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.568148 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.568153 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.568157 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.568161 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.568165 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.568170 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.568174 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.568178 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.568182 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:18 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.568186 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.568190 | controller | 16:10:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.568195 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:19 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.568199 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.568203 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.568207 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.568211 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.568216 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.568220 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.568224 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.568228 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.568232 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.568236 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.568241 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.568245 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.568250 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.568255 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.568260 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.568266 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.568272 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.568277 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.568283 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.568300 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.568304 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.568309 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.568315 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.568320 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.568324 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.568328 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.568332 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.568336 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.568340 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.568344 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.568349 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.568353 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.568357 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.568361 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.568365 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.568370 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.568375 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:19 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.568380 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.568384 | controller | 16:10:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.568390 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:20 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.568396 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.568401 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.568407 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.568416 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.568420 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.568425 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.568429 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.568433 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.568437 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.568441 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.568446 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.568450 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.568457 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.568461 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.568465 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.568469 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.568473 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.568478 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.568482 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.568488 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.568492 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.568506 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.568510 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.568514 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.568519 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.568523 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.568527 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.568531 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.568535 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.568539 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.568544 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.568548 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.568552 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.568556 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.568560 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.568564 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.568569 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:20 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.568573 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.568577 | controller | 16:10:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.568581 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:21 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.568585 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.568591 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.568596 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.568601 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.568606 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.568610 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.568614 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.568618 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.568622 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.568627 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.568631 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.568635 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.568639 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.568643 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.568647 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.568652 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.568656 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.568660 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.568664 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.568668 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.568673 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.568677 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.568688 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.568705 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.568712 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.568716 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.568720 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.568724 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.568728 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.568733 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.568737 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.568741 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.568745 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.568749 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.568754 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.568758 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.568763 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:21 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.568768 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.568772 | controller | 16:10:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.568776 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:22 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.568780 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.568785 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.568789 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.568793 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.568797 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.568801 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.568806 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.568810 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.568814 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.568818 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.568822 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.568826 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.568831 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.568835 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.568841 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.568845 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.568849 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.568853 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.568857 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.568862 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.568867 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.568872 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.568876 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.568887 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.568892 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.568896 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.568900 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.568904 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.568910 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.568914 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.568918 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.568922 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.568927 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.568931 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.568935 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.568939 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.568943 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:22 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.568957 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.568962 | controller | 16:10:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.568966 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:23 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.568970 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.568974 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.568978 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.568983 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.568989 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.568993 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.568997 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.569001 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.569005 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.569010 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.569014 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.569026 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.569030 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.569034 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.569038 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.569042 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.569046 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.569051 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.569055 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.569060 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.569065 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.569069 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.569073 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.569077 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.569082 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.569092 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.569105 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.569109 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.569113 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.569118 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.569122 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.569126 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.569130 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.569134 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.569139 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.569143 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.569147 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:23 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.569151 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.569155 | controller | 16:10:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.569160 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:25 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.569172 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.569177 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.569181 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.569185 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.569189 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.569193 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.569198 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.569202 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.569206 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.569210 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.569214 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.569218 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.569224 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.569228 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.569232 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.569237 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.569241 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.569245 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.569249 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.569253 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.569258 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.569264 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.569269 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.569275 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.569280 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.569286 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.569299 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.569304 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.569308 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.569315 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.569319 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.569323 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.569327 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.569331 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.569336 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.569340 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.569344 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:25 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.569348 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.569352 | controller | 16:10:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.569357 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:26 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.569361 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.569365 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.569369 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.569376 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.569381 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.569387 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.569392 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.569398 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.569402 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.569407 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.569411 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.569415 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.569419 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.569423 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.569428 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.569435 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.569439 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.569444 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.569449 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.569457 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.569462 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.569468 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.569472 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.569476 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.569481 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.569485 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.569489 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.569493 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.569506 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.569510 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.569514 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.569519 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.569523 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.569527 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.569531 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.569535 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.569541 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:26 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.569546 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.569552 | controller | 16:10:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.569567 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:27 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.569574 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.569579 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.569584 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.569588 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.569593 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.569597 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.569601 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.569605 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.569609 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.569614 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.569618 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.569622 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.569626 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.569630 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.569634 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.569639 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.569643 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.569647 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.569651 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.569656 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.569661 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.569666 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.569672 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.569678 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.569683 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.569688 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.569710 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.569718 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.569725 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.569737 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.569742 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.569746 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.569750 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.569754 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.569758 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.569763 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.569767 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:27 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.569779 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.569783 | controller | 16:10:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.569790 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:28 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.569795 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.569799 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.569803 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.569809 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.569813 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.569817 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.569821 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.569825 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.569830 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.569834 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.569838 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.569842 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.569846 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.569850 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.569855 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.569859 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.569863 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.569867 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.569871 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.569877 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.569882 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.569886 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.569890 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.569894 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.569898 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.569903 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.569907 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.569911 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.569915 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.569919 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.569931 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.569936 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.569940 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.569945 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.569949 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.569953 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.569957 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:28 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.569962 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.569966 | controller | 16:10:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.569970 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:29 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.569974 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.569978 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.569982 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.569987 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.569991 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.569995 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.569999 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.570003 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.570007 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.570012 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.570016 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.570020 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.570027 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.570033 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.570037 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.570041 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.570045 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.570049 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.570054 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.570058 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.570062 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.570066 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.570070 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.570074 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.570079 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.570083 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.570087 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.570091 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.570095 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.570099 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.570104 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.570108 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.570119 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.570124 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.570128 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.570132 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.570137 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:29 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.570141 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.570145 | controller | 16:10:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.570149 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:30 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.570153 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.570158 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.570162 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.570169 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.570173 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.570177 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.570181 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.570186 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.570190 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.570194 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.570198 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.570202 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.570206 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.570210 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.570216 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.570220 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.570225 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.570229 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.570233 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.570237 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.570241 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.570246 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.570250 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.570254 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.570258 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.570262 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.570266 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.570271 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.570275 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.570279 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.570283 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.570287 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.570292 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.570303 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.570308 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.570312 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.570318 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:30 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.570324 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.570329 | controller | 16:10:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.570334 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:31 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.570338 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.570342 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.570346 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.570350 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.570354 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.570359 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.570363 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.570367 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.570371 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.570376 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.570380 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.570384 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.570388 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.570392 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.570396 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.570401 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.570405 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.570409 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.570413 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.570419 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.570423 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.570427 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.570432 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.570436 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.570440 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.570444 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.570448 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.570452 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.570458 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.570462 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.570466 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.570471 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.570475 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.570479 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.570491 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.570495 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.570499 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:31 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.570504 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.570508 | controller | 16:10:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.570512 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:33 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.570516 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.570520 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.570525 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.570529 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.570533 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.570537 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.570541 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.570546 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.570550 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.570554 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.570558 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.570563 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.570570 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.570582 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.570587 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.570591 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.570595 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.570599 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.570604 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.570610 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.570615 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.570619 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.570623 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.570627 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.570631 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.570636 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.570640 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.570652 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.570656 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.570662 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.570666 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.570670 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.570675 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.570679 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.570683 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.570687 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.570707 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:33 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.570722 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.570727 | controller | 16:10:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.570731 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:34 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.570735 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.570739 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.570744 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.570748 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.570752 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.570756 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.570760 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.570764 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.570768 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.570773 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.570777 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.570781 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.570787 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.570791 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.570795 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.570800 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.570804 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.570808 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.570812 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.570816 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.570820 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.570825 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.570829 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.570833 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.570837 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.570841 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.570845 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.570850 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.570854 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.570858 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.570862 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.570866 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.570870 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.570875 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.570881 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.570885 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.570889 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:34 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.570901 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.570905 | controller | 16:10:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.570910 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:35 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.570914 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.570918 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.570922 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.570928 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.570933 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.570937 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.570941 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.570945 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.570949 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.570953 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.570958 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.570962 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.570966 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.570970 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.570974 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.570979 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.570983 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.570987 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.570991 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.570995 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.570999 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.571004 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.571008 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.571012 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.571016 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.571020 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.571024 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.571028 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.571033 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.571037 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.571041 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.571045 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.571049 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.571054 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.571058 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.571062 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.571068 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:35 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.571072 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.571084 | controller | 16:10:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.571089 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:36 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.571094 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.571098 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.571102 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.571106 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.571110 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.571115 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.571119 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.571123 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.571127 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.571132 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.571143 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.571149 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.571153 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.571158 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.571162 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.571166 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.571170 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.571174 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.571178 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.571183 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.571187 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.571191 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.571195 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.571199 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.571204 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.571208 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.571212 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.571216 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.571223 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.571227 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.571231 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.571236 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.571240 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.571244 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.571248 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.571252 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.571257 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:36 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.571262 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.571267 | controller | 16:10:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.571271 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:37 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.571282 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.571288 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.571292 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.571296 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.571300 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.571305 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.571309 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.571313 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.571317 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.571321 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.571325 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.571330 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.571334 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.571338 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.571342 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.571346 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.571351 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.571355 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.571359 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.571367 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.571372 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.571376 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.571380 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.571384 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.571388 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.571392 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.571396 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.571401 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.571405 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.571409 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.571413 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.571419 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.571423 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.571427 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.571432 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.571436 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.571440 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:37 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.571444 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.571448 | controller | 16:10:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.571453 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:38 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.571548 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.571562 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.571567 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.571572 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.571577 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.571582 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.571587 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.571592 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.571597 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.571602 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.571607 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.571612 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.571619 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.571624 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.571632 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.571637 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.571642 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.571647 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.571652 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.571657 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.571662 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.571667 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.571672 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.571676 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.571681 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.571686 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.571691 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.571714 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.571719 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.571723 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.571727 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.571732 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.571736 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.571740 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.571744 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.571748 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.571753 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:38 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.571757 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.571761 | controller | 16:10:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.571765 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:39 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.571769 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.571773 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.571777 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.571791 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.571795 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.571799 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.571804 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.571808 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.571812 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.571816 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.571822 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.571826 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.571831 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.571835 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.571839 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.571843 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.571847 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.571851 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.571856 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.571860 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.571864 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.571868 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.571872 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.571876 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.571881 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.571885 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.571889 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.571893 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.571897 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.571901 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.571906 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.571910 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.571914 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.571918 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.571922 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.571926 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.571932 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:39 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.571937 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.571941 | controller | 16:10:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.571945 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:41 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.571949 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.571953 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.571957 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.571962 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.571973 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.571977 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.571981 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.571986 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.571990 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.571994 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.571998 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.572002 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.572006 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.572011 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.572015 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.572019 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.572025 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.572029 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.572034 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.572038 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.572042 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.572046 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.572050 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.572054 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.572059 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.572063 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.572067 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.572073 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.572078 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.572084 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.572088 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.572092 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.572096 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.572100 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.572104 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.572109 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.572170 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:41 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.572176 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.572180 | controller | 16:10:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.572184 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:42 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.572188 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.572193 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.572197 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.572201 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.572205 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.572217 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.572221 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.572225 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.572230 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.572234 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.572238 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.572242 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.572246 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.572250 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.572255 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.572259 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.572263 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.572267 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.572271 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.572278 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.572282 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.572286 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.572290 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.572294 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.572299 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.572303 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.572307 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.572311 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.572315 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.572319 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.572324 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.572328 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.572332 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.572336 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.572340 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.572344 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.572348 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:42 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.572362 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.572366 | controller | 16:10:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.572370 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:43 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.572374 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.572378 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.572383 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.572387 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.572391 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.572395 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.572399 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.572410 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.572415 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.572421 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.572425 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.572429 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.572443 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.572447 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.572451 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.572455 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.572459 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.572464 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.572468 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.572473 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.572478 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.572482 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.572486 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.572490 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.572494 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.572499 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.572503 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.572507 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.572511 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.572522 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.572527 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.572531 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.572535 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.572539 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.572543 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.572548 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.572552 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:43 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.572556 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.572562 | controller | 16:10:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.572566 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:44 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.572570 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.572574 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.572579 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.572586 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.572590 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.572594 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.572598 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.572602 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.572613 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.572618 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.572622 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.572626 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.572630 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.572635 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.572639 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.572643 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.572647 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.572651 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.572656 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.572660 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.572664 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.572668 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.572672 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.572676 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.572681 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.572685 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.572705 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.572712 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.572717 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.572721 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.572725 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.572729 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.572733 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.572737 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.572742 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.572746 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.572752 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:44 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.572756 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.572760 | controller | 16:10:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.572764 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:45 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.572769 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.572773 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.572777 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.572781 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.572785 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.572789 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.572793 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.572798 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.572802 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.572806 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.572818 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.572822 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.572826 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.572830 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.572835 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.572839 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.572843 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.572847 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.572852 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.572856 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.572860 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.572864 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.572868 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.572872 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.572877 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.572881 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.572885 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.572889 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.572895 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.572899 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.572904 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.572908 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.572912 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.572916 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.572921 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.572925 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.572929 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:45 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.572933 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.572937 | controller | 16:10:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.572942 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:46 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.572946 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.572950 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.572954 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.572959 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.572963 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.572975 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.572980 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.572984 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.572990 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.572994 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.572998 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.573009 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.573013 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.573018 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.573023 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.573028 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.573032 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.573036 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.573040 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.573046 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.573050 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.573054 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.573058 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.573063 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.573067 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.573071 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.573075 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.573079 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.573084 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.573088 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.573092 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.573096 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.573100 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.573104 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.573109 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.573113 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.573117 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:46 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.573121 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.573125 | controller | 16:10:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.573129 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:47 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.573134 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.573138 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.573142 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.573146 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.573150 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.573155 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.573159 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.573163 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.573167 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.573171 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.573175 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.573180 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.573185 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.573196 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.573201 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.573205 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.573209 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.573213 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.573218 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.573222 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.573226 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.573230 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.573234 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.573238 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.573244 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.573249 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.573253 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.573257 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.573261 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.573265 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.573269 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.573274 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.573278 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.573282 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.573286 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.573290 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.573294 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:47 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.573299 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.573303 | controller | 16:10:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.573307 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:49 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.573311 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.573315 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.573320 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.573325 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.573330 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.573334 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.573338 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.573342 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.573346 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.573350 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.573354 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.573358 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.573363 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.573367 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.573378 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.573382 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.573386 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.573390 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.573395 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.573400 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.573405 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.573409 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.573413 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.573417 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.573421 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.573425 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.573429 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.573434 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.573438 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.573443 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.573448 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.573452 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.573456 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.573460 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.573464 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.573469 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.573474 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:49 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.573478 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.573483 | controller | 16:10:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.573487 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:50 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.573491 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.573495 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.573499 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.573503 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.573509 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.573513 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.573518 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.573522 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.573526 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.573530 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.573534 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.573539 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.573543 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.573547 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.573551 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.573563 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.573567 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.573571 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.573576 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.573580 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.573584 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.573588 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.573592 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.573597 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.573601 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.573605 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.573609 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.573613 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.573619 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.573623 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.573627 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.573632 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.573636 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.573640 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.573644 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.573648 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.573653 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:50 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.573657 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.573661 | controller | 16:10:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.573665 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:51 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.573670 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.573674 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.573678 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.573682 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.573686 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.573691 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.573708 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.573712 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.573716 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.573720 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.573725 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.573729 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.573733 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.573737 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.573741 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.573746 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.573750 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.573761 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.573765 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.573773 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.573777 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.573782 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.573786 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.573792 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.573796 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.573800 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.573804 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.573808 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.573812 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.573816 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.573821 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.573825 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.573829 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.573833 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.573837 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.573842 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.573846 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:51 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.573850 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.573854 | controller | 16:10:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.573858 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:52 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.573863 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.573867 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.573871 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.573877 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.573889 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.573894 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.573898 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.573902 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.573906 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.573910 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.573914 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.573918 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.573924 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.573928 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.573932 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.573936 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.573941 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.573945 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.573963 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.573968 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.573972 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.573976 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.573980 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.573984 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.573988 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.573993 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.573997 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.574001 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.574005 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.574009 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.574014 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.574018 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.574022 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.574026 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.574030 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.574034 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.574049 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:52 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.574054 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.574058 | controller | 16:10:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.574062 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:53 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.574066 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.574070 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.574076 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.574082 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.574086 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.574091 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.574095 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.574099 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.574103 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.574107 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.574111 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.574115 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.574120 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.574124 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.574128 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.574132 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.574143 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.574148 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.574152 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.574161 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.574166 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.574170 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.574174 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.574178 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.574183 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.574187 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.574191 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.574195 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.574206 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.574211 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.574215 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.574219 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.574223 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.574227 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.574232 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.574236 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.574241 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:53 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.574246 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.574250 | controller | 16:10:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.574254 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:54 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.574258 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.574262 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.574266 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.574271 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.574275 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.574279 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.574283 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.574287 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.574291 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.574296 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.574300 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.574304 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.574308 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.574312 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.574320 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.574325 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.574329 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.574334 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.574339 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.574346 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.574356 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.574360 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.574366 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.574370 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.574374 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.574378 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.574382 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.574386 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.574392 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.574396 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.574401 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.574405 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.574409 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.574413 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.574417 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.574422 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.574426 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:54 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.574430 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.574434 | controller | 16:10:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.574438 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:55 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.574442 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.574447 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.574451 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.574455 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.574459 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.574463 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.574467 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.574472 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.574476 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.574480 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.574484 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.574488 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.574492 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.574497 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.574501 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.574505 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.574509 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.574513 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.574517 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.574532 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.574536 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.574540 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.574551 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.574555 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.574559 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.574563 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.574568 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.574572 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.574576 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.574580 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.574584 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.574588 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.574593 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.574597 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.574601 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.574605 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.574609 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:55 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.574613 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.574618 | controller | 16:10:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.574622 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:57 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.574626 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.574632 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.574636 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.574640 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.574644 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.574649 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.574653 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.574657 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.574661 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.574665 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.574669 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.574674 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.574679 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.574684 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.574688 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.574702 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.574709 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.574713 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.574717 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.574721 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.574725 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.574730 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.574734 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.574744 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.574749 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.574760 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.574764 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.574769 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.574773 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.574783 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.574787 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.574791 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.574796 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.574800 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.574804 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.574808 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.574812 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:57 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.574816 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.574821 | controller | 16:10:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.574825 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:58 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.574829 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.574833 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.574837 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.574845 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.574849 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.574853 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.574857 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.574861 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.574866 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.574870 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.574874 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.574886 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.574890 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.574895 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.574899 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.574903 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.574907 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.574911 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.574916 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.574920 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.574926 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.574930 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.574934 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.574938 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.574943 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.574952 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.574957 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.574961 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.574966 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.574970 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.574974 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.574978 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.574982 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.574987 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.574991 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.574995 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.575001 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:58 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.575005 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.575009 | controller | 16:10:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.575013 | controller | \"ac-watcher\" not found\n logger.go:42: 16:10:59 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.575018 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.575022 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.575026 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.575030 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.575034 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.575038 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.575043 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.575047 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.575051 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.575055 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.575059 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.575063 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.575067 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.575072 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.575076 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.575080 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.575084 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.575088 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.575092 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.575097 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.575101 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.575105 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.575109 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.575113 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.575117 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.575121 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.575132 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.575136 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.575142 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.575146 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.575151 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.575155 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.575159 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.575163 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.575167 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.575171 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.575176 | controller | expected application_credential_id\"]\n logger.go:42: 16:10:59 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.575180 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.575184 | controller | 16:10:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.575188 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:00 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.575194 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.575198 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.575202 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.575208 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.575212 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.575216 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.575221 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.575225 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.575229 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.575233 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.575237 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.575241 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.575246 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.575250 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.575254 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.575258 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.575262 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.575266 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.575271 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.575278 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.575282 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.575286 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.575290 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.575295 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.575299 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.575303 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.575307 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.575311 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.575322 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.575327 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.575331 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.575335 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.575339 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.575344 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.575348 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.575352 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.575356 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:00 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.575360 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.575364 | controller | 16:11:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.575369 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:01 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.575373 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.575377 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.575381 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.575385 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.575390 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.575394 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.575398 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.575402 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.575406 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.575410 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.575415 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.575419 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.575424 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.575429 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.575433 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.575439 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.575443 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.575447 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.575451 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.575455 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.575460 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.575477 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.575481 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.575485 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.575489 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.575493 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.575497 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.575502 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.575506 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.575516 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.575520 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.575525 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.575529 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.575533 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.575537 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.575541 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.575546 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:01 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.575550 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.575554 | controller | 16:11:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.575558 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:02 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.575562 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.575566 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.575571 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.575576 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.575581 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.575585 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.575589 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.575593 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.575598 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.575602 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.575606 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.575610 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.575614 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.575618 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.575624 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.575628 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.575633 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.575637 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.575641 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.575645 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.575649 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.575654 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.575658 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.575662 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.575666 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.575670 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.575675 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.575679 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.575683 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.575687 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.575709 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.575716 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.575720 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.575725 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.575729 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.575733 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.575739 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:02 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.575745 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.575749 | controller | 16:11:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.575755 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:04 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.575759 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.575763 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.575767 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.575772 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.575776 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.575780 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.575784 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.575788 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.575792 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.575796 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.575801 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.575805 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.575809 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.575813 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.575817 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.575822 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.575826 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.575830 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.575834 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.575838 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.575842 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.575847 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.575851 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.575855 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.575859 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.575863 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.575867 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.575871 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.575878 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.575883 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.575887 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.575891 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.575902 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.575906 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.575911 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.575915 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.575919 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:04 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.575923 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.575927 | controller | 16:11:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.575932 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:05 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.575936 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.575940 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.575944 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.575948 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.575952 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.575957 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.575961 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.575965 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.575971 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.575975 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.575979 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.575983 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.575988 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.575993 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.575997 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.576001 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.576005 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.576010 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.576014 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.576020 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.576024 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.576028 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.576032 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.576037 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.576041 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.576045 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.576049 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.576053 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.576069 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.576075 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.576080 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.576084 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.576088 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.576098 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.576103 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.576107 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.576111 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:05 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.576115 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.576119 | controller | 16:11:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.576124 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:06 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.576128 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.576132 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.576136 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.576140 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.576152 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.576156 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.576160 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.576165 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.576169 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.576173 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.576177 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.576181 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.576187 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.576191 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.576195 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.576200 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.576204 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.576208 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.576212 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.576218 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.576223 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.576238 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.576242 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.576246 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.576250 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.576254 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.576259 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.576263 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.576267 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.576271 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.576275 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.576280 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.576284 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.576288 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.576298 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.576303 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.576307 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:06 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.576313 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.576317 | controller | 16:11:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.576321 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:07 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.576325 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.576329 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.576334 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.576339 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.576344 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.576348 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.576352 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.576356 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.576360 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.576364 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.576368 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.576373 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.576377 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.576381 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.576385 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.576389 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.576394 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.576398 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.576402 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.576406 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.576410 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.576414 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.576419 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.576423 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.576427 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.576431 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.576435 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.576439 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.576444 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.576448 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.576452 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.576456 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.576460 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.576464 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.576469 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.576473 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.576484 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:07 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.576489 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.576493 | controller | 16:11:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.576497 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:08 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.576501 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.576505 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.576509 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.576515 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.576519 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.576524 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.576528 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.576532 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.576536 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.576540 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.576544 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.576549 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.576554 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.576558 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.576562 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.576567 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.576571 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.576575 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.576579 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.576583 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.576587 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.576592 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.576596 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.576600 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.576604 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.576608 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.576612 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.576617 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.576631 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.576635 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.576639 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.576643 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.576648 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.576652 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.576656 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.576660 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.576664 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:08 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.576674 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.576678 | controller | 16:11:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.576683 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:09 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.576687 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.576701 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.576708 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.576712 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.576716 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.576720 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.576725 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.576729 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.576733 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.576737 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.576741 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.576745 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.576749 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.576754 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.576758 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.576762 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.576766 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.576770 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.576774 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.576780 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.576784 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.576789 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.576793 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.576797 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.576801 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.576805 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.576809 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.576814 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.576818 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.576822 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.576826 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.576830 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.576835 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.576839 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.576845 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.576849 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.576853 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:09 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.576857 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.576868 | controller | 16:11:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.576873 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:11 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.576877 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.576881 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.576886 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.576890 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.576894 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.576898 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.576903 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.576907 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.576911 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.576915 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.576919 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.576923 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.576930 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.576935 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.576943 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.576950 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.576956 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.576962 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.576968 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.576973 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.576979 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.576985 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.576990 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.576994 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.576998 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.577002 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.577007 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.577011 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.577015 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.577019 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.577023 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.577028 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.577032 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.577036 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.577040 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.577045 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.577049 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:11 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.577053 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.577057 | controller | 16:11:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.577069 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:12 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.577073 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.577078 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.577082 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.577088 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.577093 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.577097 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.577101 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.577105 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.577109 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.577114 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.577118 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.577124 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.577128 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.577133 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.577137 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.577141 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.577145 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.577149 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.577154 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.577159 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.577164 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.577168 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.577172 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.577176 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.577180 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.577185 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.577189 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.577193 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.577197 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.577201 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.577206 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.577210 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.577214 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.577218 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.577223 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.577227 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.577232 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:12 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.577237 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.577241 | controller | 16:11:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.577245 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:13 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.577249 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.577260 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.577264 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.577269 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.577273 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.577277 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.577281 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.577286 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.577291 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.577296 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.577301 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.577305 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.577310 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.577315 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.577320 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.577324 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.577328 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.577333 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.577337 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.577341 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.577345 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.577349 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.577355 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.577360 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.577364 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.577368 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.577372 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.577376 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.577382 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.577388 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.577392 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.577398 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.577402 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.577406 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.577410 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.577414 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.577419 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:13 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.577423 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.577427 | controller | 16:11:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.577432 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:14 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.577436 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.577440 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.577451 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.577455 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.577460 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.577464 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.577468 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.577472 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.577476 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.577481 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.577485 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.577489 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.577493 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.577497 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.577502 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.577506 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.577510 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.577514 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.577518 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.577524 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.577529 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.577533 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.577537 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.577541 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.577546 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.577550 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.577554 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.577558 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.577562 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.577567 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.577571 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.577576 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.577581 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.577587 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.577593 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.577598 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.577604 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:14 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.577609 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.577614 | controller | 16:11:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.577618 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:15 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.577622 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.577626 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.577631 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.577635 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.577646 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.577651 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.577655 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.577659 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.577663 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.577668 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.577674 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.577678 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.577684 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.577688 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.577706 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.577713 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.577719 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.577725 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.577730 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.577736 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.577742 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.577748 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.577754 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.577758 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.577763 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.577767 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.577771 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.577775 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.577780 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.577784 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.577788 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.577792 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.577796 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.577801 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.577805 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.577809 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.577813 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:15 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.577818 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.577822 | controller | 16:11:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.577826 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:16 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.577830 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.577835 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.577839 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.577847 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.577854 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.577866 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.577871 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.577876 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.577880 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.577884 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.577888 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.577892 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.577897 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.577901 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.577905 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.577909 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.577914 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.577918 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.577922 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.577926 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.577930 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.577935 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.577939 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.577943 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.577947 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.577951 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.577956 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.577960 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.577966 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.577974 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.577980 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.577986 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.577992 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.577998 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.578002 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.578008 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.578016 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:16 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.578022 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.578028 | controller | 16:11:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.578034 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:17 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.578039 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.578054 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.578058 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.578062 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.578067 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.578071 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.578083 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.578087 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.578091 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.578096 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.578101 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.578107 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.578112 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.578127 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.578132 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.578136 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.578140 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.578145 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.578149 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.578155 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.578160 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.578164 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.578168 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.578172 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.578177 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.578181 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.578193 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.578197 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.578203 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.578208 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.578212 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.578216 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.578220 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.578225 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.578229 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.578233 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.578237 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:17 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.578242 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.578248 | controller | 16:11:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.578254 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:19 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.578260 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.578265 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.578269 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.578274 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.578278 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.578282 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.578286 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.578290 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.578302 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.578311 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.578317 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.578323 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.578328 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.578333 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.578338 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.578343 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.578347 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.578351 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.578356 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.578364 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.578370 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.578376 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.578382 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.578388 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.578392 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.578397 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.578401 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.578405 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.578409 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.578414 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.578418 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.578422 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.578426 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.578430 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.578435 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.578440 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.578445 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:19 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.578451 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.578456 | controller | 16:11:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.578462 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:20 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.578466 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.578470 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.578475 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.578479 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.578483 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.578487 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.578492 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.578496 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.578500 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.578512 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.578518 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.578522 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.578528 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.578552 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.578558 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.578564 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.578570 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.578575 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.578581 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.578586 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.578590 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.578595 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.578599 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.578603 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.578607 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.578611 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.578618 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.578622 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.578626 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.578631 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.578635 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.578639 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.578643 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.578647 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.578652 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.578656 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.578660 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:20 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.578664 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.578669 | controller | 16:11:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.578674 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:21 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.578679 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.578685 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.578691 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.578723 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.578729 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.578734 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.578738 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.578743 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.578747 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.578751 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.578755 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.578766 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.578771 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.578775 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.578779 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.578784 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.578788 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.578792 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.578796 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.578800 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.578805 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.578809 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.578813 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.578817 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.578821 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.578825 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.578829 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.578834 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.578838 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.578844 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.578848 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.578853 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.578857 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.578861 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.578865 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.578869 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.578876 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:21 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.578880 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.578884 | controller | 16:11:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.578889 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:22 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.578893 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.578897 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.578901 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.578905 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.578909 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.578914 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.578918 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.578922 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.578928 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.578932 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.578937 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.578943 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.578956 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.578962 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.578967 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.578971 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.578975 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.578979 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.578983 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.578988 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.578992 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.578996 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.579000 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.579004 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.579009 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.579013 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.579017 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.579021 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.579027 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.579033 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.579038 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.579044 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.579050 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.579056 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.579061 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.579067 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.579072 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:22 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.579078 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.579083 | controller | 16:11:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.579088 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:23 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.579092 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.579096 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.579100 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.579104 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.579109 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.579113 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.579117 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.579122 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.579128 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.579133 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.579139 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.579145 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.579150 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.579154 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.579167 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.579172 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.579176 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.579181 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.579185 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.579195 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.579199 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.579203 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.579207 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.579212 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.579217 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.579221 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.579226 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.579230 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.579234 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.579238 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.579242 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.579246 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.579251 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.579257 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.579262 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.579268 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.579274 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:23 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.579279 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.579284 | controller | 16:11:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.579289 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:24 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.579293 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.579297 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.579301 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.579307 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.579312 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.579316 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.579320 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.579324 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.579328 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.579332 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.579337 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.579341 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.579348 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.579352 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.579356 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.579368 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.579374 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.579378 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.579382 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.579387 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.579391 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.579395 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.579399 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.579403 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.579407 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.579412 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.579416 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.579420 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.579424 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.579428 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.579432 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.579437 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.579441 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.579445 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.579449 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.579454 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.579458 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:24 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.579462 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.579487 | controller | 16:11:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.579492 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:26 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.579496 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.579502 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.579507 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.579515 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.579521 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.579527 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.579532 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.579539 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.579543 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.579548 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.579552 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.579556 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.579560 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.579565 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.579569 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.579573 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.579584 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.579589 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.579593 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.579597 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.579602 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.579606 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.579610 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.579614 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.579618 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.579623 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.579627 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.579631 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.579635 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.579639 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.579643 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.579647 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.579652 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.579656 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.579660 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.579664 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.579673 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:26 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.579677 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.579681 | controller | 16:11:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.579686 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:27 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.579690 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.579709 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.579714 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.579718 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.579722 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.579726 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.579731 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.579735 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.579739 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.579743 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.579747 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.579752 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.579756 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.579760 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.579766 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.579770 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.579775 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.579779 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.579790 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.579795 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.579799 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.579803 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.579808 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.579813 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.579818 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.579822 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.579826 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.579830 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.579836 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.579841 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.579845 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.579849 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.579853 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.579857 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.579862 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.579866 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.579870 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:27 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.579874 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.579878 | controller | 16:11:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.579883 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:28 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.579887 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.579891 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.579895 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.579900 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.579904 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.579908 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.579912 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.579916 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.579920 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.579925 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.579929 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.579933 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.579937 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.579941 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.579946 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.579950 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.579954 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.579958 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.579963 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.579975 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.579979 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.579984 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.579988 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.579992 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.579996 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.580000 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.580005 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.580009 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.580013 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.580017 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.580021 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.580026 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.580030 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.580034 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.580038 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.580042 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.580047 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:28 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.580051 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.580055 | controller | 16:11:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.580059 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:29 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.580063 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.580067 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.580073 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.580077 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.580082 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.580086 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.580090 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.580094 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.580098 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.580103 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.580107 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.580111 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.580117 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.580121 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.580125 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.580129 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.580134 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.580138 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.580142 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.580148 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.580159 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.580163 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.580168 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.580172 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.580176 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.580180 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.580184 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.580189 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.580193 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.580200 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.580204 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.580208 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.580213 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.580217 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.580221 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.580225 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.580229 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:29 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.580235 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.580239 | controller | 16:11:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.580243 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:30 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.580247 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.580252 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.580256 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.580261 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.580266 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.580270 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.580274 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.580278 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.580282 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.580287 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.580291 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.580295 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.580299 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.580303 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.580308 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.580312 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.580316 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.580320 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.580324 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.580329 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.580333 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.580337 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.580349 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.580354 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.580358 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.580362 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.580367 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.580371 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.580375 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.580379 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.580383 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.580387 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.580391 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.580396 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.580400 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.580404 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.580410 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:30 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.580414 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.580418 | controller | 16:11:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.580423 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:31 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.580427 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.580431 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.580435 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.580439 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.580443 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.580447 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.580452 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.580456 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.580460 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.580464 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.580468 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.580472 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.580477 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.580481 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.580485 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.580489 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.580493 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.580498 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.580502 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.580506 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.580510 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.580514 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.580518 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.580529 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.580535 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.580539 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.580543 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.580547 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.580553 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.580557 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.580561 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.580566 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.580570 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.580574 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.580578 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.580582 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.580586 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:31 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.580591 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.580595 | controller | 16:11:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.580599 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:33 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.580603 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.580609 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.580613 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.580619 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.580623 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.580627 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.580631 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.580635 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.580639 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.580643 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.580648 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.580652 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.580656 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.580660 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.580664 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.580668 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.580673 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.580677 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.580681 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.580686 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.580691 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.580712 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.580716 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.580720 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.580725 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.580735 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.580739 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.580744 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.580748 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.580752 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.580756 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.580761 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.580766 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.580772 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.580777 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.580783 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.580789 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:33 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.580794 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.580798 | controller | 16:11:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.580803 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:34 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.580807 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.580811 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.580815 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.580819 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.580823 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.580828 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.580832 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.580836 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.580841 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.580847 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.580852 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.580858 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.580866 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.580872 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.580878 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.580883 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.580887 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.580891 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.580895 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.580899 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.580906 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.580910 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.580914 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.580918 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.580923 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.580927 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.580938 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.580944 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.580949 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.580955 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.580960 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.580966 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.580972 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.580977 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.580981 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.580985 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.580989 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:34 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.580993 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.580998 | controller | 16:11:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.581002 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:35 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.581006 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.581010 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.581014 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.581021 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.581025 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.581029 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.581033 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.581038 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.581042 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.581046 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.581050 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.581054 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.581058 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.581063 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.581069 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.581073 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.581077 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.581081 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.581085 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.581091 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.581095 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.581099 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.581103 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.581108 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.581112 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.581116 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.581120 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.581124 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.581136 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.581140 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.581145 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.581149 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.581153 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.581157 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.581161 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.581166 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.581173 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:35 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.581177 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.581181 | controller | 16:11:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.581185 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:36 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.581191 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.581195 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.581200 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.581204 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.581208 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.581212 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.581216 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.581220 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.581225 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.581229 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.581233 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.581237 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.581241 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.581246 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.581250 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.581255 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.581260 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.581266 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.581271 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.581277 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.581282 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.581286 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.581290 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.581294 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.581298 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.581303 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.581307 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.581311 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.581317 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.581328 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.581332 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.581337 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.581341 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.581345 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.581349 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.581354 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.581359 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:36 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.581365 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.581370 | controller | 16:11:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.581376 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:37 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.581381 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.581385 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.581389 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.581393 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.581397 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.581402 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.581406 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.581410 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.581414 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.581418 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.581423 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.581427 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.581431 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.581435 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.581439 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.581445 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.581450 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.581454 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.581458 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.581464 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.581468 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.581472 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.581476 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.581480 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.581484 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.581489 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.581493 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.581497 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.581501 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.581506 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.581518 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.581522 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.581526 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.581531 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.581535 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.581539 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.581543 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:37 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.581547 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.581551 | controller | 16:11:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.581556 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:38 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.581560 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.581564 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.581568 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.581572 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.581576 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.581580 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.581585 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.581589 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.581593 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.581597 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.581601 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.581605 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.581611 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.581615 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.581619 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.581624 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.581628 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.581632 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.581636 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.581640 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.581644 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.581649 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.581653 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.581657 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.581661 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.581665 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.581669 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.581674 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.581678 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.581682 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.581686 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.581690 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.581716 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.581721 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.581725 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.581729 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.581734 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:38 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.581738 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.581742 | controller | 16:11:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.581748 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:39 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.581752 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.581756 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.581761 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.581766 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.581771 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.581775 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.581779 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.581783 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.581787 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.581792 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.581796 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.581800 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.581804 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.581808 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.581812 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.581817 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.581821 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.581825 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.581829 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.581833 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.581838 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.581842 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.581846 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.581850 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.581854 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.581858 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.581863 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.581867 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.581871 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.581875 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.581879 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.581883 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.581887 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.581898 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.581903 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.581907 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.581913 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:39 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.581917 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.581921 | controller | 16:11:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.581925 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:41 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.581929 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.581934 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.581938 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.581943 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.581948 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.581952 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.581956 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.581960 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.581964 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.581968 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.581972 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.581977 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.581981 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.581986 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.581991 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.581995 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.581999 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.582003 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.582007 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.582013 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.582017 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.582021 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.582025 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.582029 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.582034 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.582038 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.582042 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.582046 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.582052 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.582056 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.582060 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.582064 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.582068 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.582073 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.582083 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.582087 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.582092 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:41 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.582096 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.582100 | controller | 16:11:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.582104 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:42 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.582108 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.582113 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.582117 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.582121 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.582125 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.582129 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.582134 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.582138 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.582142 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.582146 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.582150 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.582154 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.582159 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.582163 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.582167 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.582171 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.582175 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.582179 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.582184 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.582189 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.582193 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.582198 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.582202 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.582206 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.582210 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.582214 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.582218 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.582223 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.582227 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.582231 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.582235 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.582239 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.582243 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.582247 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.582252 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.582265 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.582270 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:42 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.582274 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.582278 | controller | 16:11:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.582282 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:43 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.582287 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.582291 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.582295 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.582299 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.582305 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.582309 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.582314 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.582319 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.582323 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.582327 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.582331 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.582335 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.582341 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.582345 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.582349 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.582355 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.582359 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.582363 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.582367 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.582372 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.582376 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.582380 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.582384 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.582389 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.582393 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.582397 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.582401 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.582405 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.582409 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.582414 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.582418 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.582422 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.582426 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.582431 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.582435 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.582439 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.582443 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:43 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.582454 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.582459 | controller | 16:11:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.582463 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:44 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.582467 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.582471 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.582476 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.582481 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.582486 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.582490 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.582494 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.582498 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.582502 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.582507 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.582511 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.582515 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.582521 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.582525 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.582529 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.582533 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.582537 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.582542 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.582546 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.582550 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.582554 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.582558 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.582562 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.582566 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.582571 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.582575 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.582579 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.582583 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.582587 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.582591 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.582595 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.582600 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.582604 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.582608 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.582612 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.582616 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.582622 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:44 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.582626 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.582636 | controller | 16:11:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.582640 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:45 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.582645 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.582649 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.582653 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.582657 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.582661 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.582665 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.582670 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.582674 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.582678 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.582682 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.582686 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.582691 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.582712 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.582717 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.582721 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.582725 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.582729 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.582733 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.582737 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.582741 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.582746 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.582750 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.582754 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.582758 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.582762 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.582766 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.582770 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.582775 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.582780 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.582786 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.582790 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.582795 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.582799 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.582803 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.582808 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.582812 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.582817 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:45 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.582821 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.582825 | controller | 16:11:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.582835 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:46 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.582840 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.582844 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.582848 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.582852 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.582857 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.582861 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.582865 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.582869 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.582873 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.582877 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.582882 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.582886 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.582890 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.582894 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.582898 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.582902 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.582907 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.582911 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.582915 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.582922 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.582926 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.582930 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.582935 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.582939 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.582943 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.582947 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.582951 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.582955 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.582959 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.582964 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.582968 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.582972 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.582976 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.582980 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.582984 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.582988 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.582993 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:46 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.582997 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.583001 | controller | 16:11:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.583005 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:48 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.583009 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.583019 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.583024 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.583028 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.583032 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.583036 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.583041 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.583045 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.583049 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.583053 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.583057 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.583063 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.583068 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.583073 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.583077 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.583081 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.583085 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.583089 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.583094 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.583098 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.583102 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.583106 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.583110 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.583114 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.583118 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.583123 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.583127 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.583131 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.583135 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.583139 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.583143 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.583148 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.583152 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.583156 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.583160 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.583164 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.583169 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:48 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.583173 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.583177 | controller | 16:11:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.583181 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:49 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.583185 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.583189 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.583200 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.583208 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.583213 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.583217 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.583221 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.583225 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.583229 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.583234 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.583238 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.583242 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.583246 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.583250 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.583254 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.583258 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.583263 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.583267 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.583271 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.583275 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.583279 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.583283 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.583288 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.583292 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.583296 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.583300 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.583304 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.583308 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.583312 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.583317 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.583321 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.583327 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.583331 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.583335 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.583339 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.583343 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.583349 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:49 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.583353 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.583357 | controller | 16:11:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.583361 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:50 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.583366 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.583370 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.583374 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.583378 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.583389 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.583393 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.583398 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.583402 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.583406 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.583410 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.583414 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.583418 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.583422 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.583427 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.583431 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.583435 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.583439 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.583443 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.583447 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.583452 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.583456 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.583460 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.583464 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.583469 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.583558 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.583564 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.583569 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.583574 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.583581 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.583586 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.583591 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.583596 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.583601 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.583606 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.583611 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.583616 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.583620 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:50 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.583625 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.583630 | controller | 16:11:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.583635 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:52 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.583640 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.583645 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.583650 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.583655 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.583660 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.583675 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.583681 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.583686 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.583691 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.583713 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.583726 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.583732 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.583736 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.583740 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.583745 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.583750 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.583754 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.583758 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.583762 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.583768 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.583772 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.583776 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.583781 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.583785 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.583789 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.583793 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.583797 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.583801 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.583806 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.583810 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.583814 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.583818 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.583822 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.583827 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.583831 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.583835 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.583839 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:52 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.583843 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.583847 | controller | 16:11:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.583852 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:53 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.583856 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.583860 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.583864 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.583868 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.583872 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.583877 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.583888 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.583893 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.583897 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.583901 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.583905 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.583909 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.583915 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.583919 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.583924 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.583928 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.583932 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.583936 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.583940 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.583946 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.583950 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.583954 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.583958 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.583962 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.583967 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.583971 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.583975 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.583980 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.583984 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.583989 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.583993 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.583997 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.584001 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.584005 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.584009 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.584014 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.584018 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:53 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.584022 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.584026 | controller | 16:11:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.584030 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:54 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.584035 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.584039 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.584043 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.584049 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.584053 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.584057 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.584061 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.584065 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.584076 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.584081 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.584085 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.584089 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.584093 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.584097 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.584102 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.584106 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.584110 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.584114 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.584118 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.584122 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.584127 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.584131 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.584135 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.584139 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.584143 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.584147 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.584152 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.584156 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.584160 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.584166 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.584170 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.584174 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.584178 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.584183 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.584187 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.584191 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.584196 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:54 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.584201 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.584205 | controller | 16:11:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.584209 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:55 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.584213 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.584217 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.584222 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.584226 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.584230 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.584234 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.584239 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.584243 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.584247 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.584260 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.584264 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.584268 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.584273 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.584277 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.584281 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.584285 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.584289 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.584294 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.584298 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.584302 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.584306 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.584310 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.584314 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.584319 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.584323 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.584327 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.584331 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.584335 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.584342 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.584346 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.584351 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.584355 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.584359 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.584363 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.584367 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.584371 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.584375 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:55 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.584380 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.584384 | controller | 16:11:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.584388 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:56 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.584392 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.584396 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.584401 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.584405 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.584409 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.584413 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.584417 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.584422 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.584426 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.584430 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.584434 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.584445 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.584449 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.584454 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.584458 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.584462 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.584466 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.584470 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.584475 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.584480 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.584485 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.584489 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.584493 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.584497 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.584501 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.584505 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.584511 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.584515 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.584520 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.584524 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.584528 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.584532 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.584536 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.584540 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.584545 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.584549 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.584553 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:56 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.584559 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.584563 | controller | 16:11:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.584567 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:57 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.584571 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.584575 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.584579 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.584585 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.584589 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.584593 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.584598 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.584602 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.584606 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.584610 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.584614 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.584618 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.584631 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.584635 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.584639 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.584643 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.584648 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.584652 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.584656 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.584660 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.584664 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.584668 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.584673 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.584677 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.584681 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.584685 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.584689 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.584706 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.584711 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.584715 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.584719 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.584723 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.584727 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.584732 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.584736 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.584740 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.584744 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:57 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.584748 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.584752 | controller | 16:11:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.584757 | controller | \"ac-watcher\" not found\n logger.go:42: 16:11:58 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.584761 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.584765 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.584769 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.584775 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.584779 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.584784 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.584788 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.584792 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.584798 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.584802 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.584806 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.584810 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.584814 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.584819 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.584830 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.584834 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.584839 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.584843 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.584847 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.584853 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.584857 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.584861 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.584865 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.584869 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.584873 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.584878 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.584882 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.584886 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.584890 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.584894 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.584898 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.584903 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.584907 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.584911 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.584915 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.584919 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.584925 | controller | expected application_credential_id\"]\n logger.go:42: 16:11:58 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.584929 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.584933 | controller | 16:11:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.584937 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:00 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.584942 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.584946 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.584950 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.584954 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.584958 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.584962 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.584967 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.584971 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.584975 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.584979 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.584983 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.584987 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.584991 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.584996 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.585001 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.585011 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.585015 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.585020 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.585024 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.585028 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.585032 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.585036 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.585040 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.585044 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.585050 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.585054 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.585059 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.585063 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.585068 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.585072 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.585077 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.585081 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.585085 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.585089 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.585093 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.585098 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.585102 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:00 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.585106 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.585110 | controller | 16:12:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.585114 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:01 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.585118 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.585123 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.585127 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.585131 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.585135 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.585139 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.585143 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.585148 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.585152 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.585156 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.585160 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.585164 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.585168 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.585173 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.585177 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.585181 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.585191 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.585195 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.585199 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.585205 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.585209 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.585213 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.585218 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.585222 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.585226 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.585230 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.585234 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.585238 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.585243 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.585247 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.585251 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.585255 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.585259 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.585263 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.585268 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.585272 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.585276 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:01 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.585280 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.585284 | controller | 16:12:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.585288 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:02 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.585293 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.585297 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.585301 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.585305 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.585309 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.585313 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.585317 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.585323 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.585327 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.585332 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.585336 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.585340 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.585347 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.585351 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.585355 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.585359 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.585364 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.585373 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.585378 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.585382 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.585386 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.585390 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.585394 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.585399 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.585403 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.585407 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.585411 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.585415 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.585419 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.585425 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.585429 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.585434 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.585438 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.585442 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.585446 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.585450 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.585454 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:02 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.585459 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.585463 | controller | 16:12:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.585467 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:03 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.585473 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.585477 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.585481 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.585487 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.585491 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.585496 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.585500 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.585504 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.585508 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.585512 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.585516 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.585520 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.585525 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.585529 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.585533 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.585537 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.585541 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.585545 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.585550 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.585560 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.585564 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.585569 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.585573 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.585578 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.585582 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.585587 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.585591 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.585595 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.585599 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.585603 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.585607 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.585611 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.585616 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.585620 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.585624 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.585628 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.585634 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:03 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.585638 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.585642 | controller | 16:12:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.585646 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:04 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.585650 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.585655 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.585659 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.585663 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.585667 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.585671 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.585675 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.585680 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.585684 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.585688 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.585702 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.585708 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.585713 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.585717 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.585721 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.585725 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.585729 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.585733 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.585737 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.585743 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.585754 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.585758 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.585762 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.585767 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.585771 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.585775 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.585779 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.585783 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.585789 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.585793 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.585797 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.585802 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.585806 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.585810 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.585814 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.585818 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.585822 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:04 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.585826 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.585831 | controller | 16:12:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.585835 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:05 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.585839 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.585843 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.585849 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.585854 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.585859 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.585863 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.585867 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.585871 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.585875 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.585880 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.585884 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.585888 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.585892 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.585896 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.585900 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.585905 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.585909 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.585913 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.585917 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.585923 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.585927 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.585937 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.585941 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.585946 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.585950 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.585954 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.585958 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.585962 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.585967 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.585971 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.585975 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.585979 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.585983 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.585987 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.585992 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.585996 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.586000 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:05 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.586004 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.586008 | controller | 16:12:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.586012 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:06 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.586017 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.586021 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.586025 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.586029 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.586033 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.586037 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.586042 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.586046 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.586050 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.586054 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.586058 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.586063 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.586068 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.586072 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.586077 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.586081 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.586085 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.586089 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.586093 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.586097 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.586102 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.586106 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.586111 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.586122 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.586126 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.586130 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.586134 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.586139 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.586143 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.586147 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.586151 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.586155 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.586159 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.586163 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.586168 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.586172 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.586176 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:06 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.586180 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.586184 | controller | 16:12:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.586189 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:08 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.586193 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.586197 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.586201 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.586207 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.586211 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.586215 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.586219 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.586223 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.586228 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.586232 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.586236 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.586240 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.586244 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.586248 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.586254 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.586258 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.586263 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.586267 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.586271 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.586275 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.586279 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.586283 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.586288 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.586292 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.586302 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.586306 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.586310 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.586315 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.586319 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.586323 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.586327 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.586331 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.586335 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.586340 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.586344 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.586348 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.586355 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:08 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.586359 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.586363 | controller | 16:12:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.586368 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:09 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.586372 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.586377 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.586382 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.586386 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.586390 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.586394 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.586398 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.586402 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.586407 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.586411 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.586415 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.586419 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.586423 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.586427 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.586431 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.586436 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.586440 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.586444 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.586448 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.586452 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.586456 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.586461 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.586465 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.586469 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.586473 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.586477 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.586487 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.586491 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.586497 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.586502 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.586506 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.586510 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.586514 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.586518 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.586522 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.586526 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.586531 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:09 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.586535 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.586539 | controller | 16:12:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.586543 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:10 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.586547 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.586551 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.586556 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.586560 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.586565 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.586570 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.586574 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.586578 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.586582 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.586586 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.586591 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.586595 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.586599 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.586603 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.586607 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.586612 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.586616 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.586620 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.586624 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.586630 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.586635 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.586639 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.586644 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.586648 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.586652 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.586656 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.586662 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.586672 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.586676 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.586681 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.586685 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.586691 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.586707 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.586712 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.586716 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.586720 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.586724 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:10 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.586728 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.586732 | controller | 16:12:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.586736 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:11 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.586741 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.586745 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.586749 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.586753 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.586757 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.586762 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.586766 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.586770 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.586774 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.586778 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.586782 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.586786 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.586792 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.586796 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.586800 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.586805 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.586809 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.586813 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.586817 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.586821 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.586825 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.586830 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.586834 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.586838 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.586842 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.586846 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.586850 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.586854 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.586858 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.586869 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.586873 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.586878 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.586882 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.586886 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.586890 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.586894 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.586898 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:11 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.586902 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.586907 | controller | 16:12:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.586911 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:12 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.586917 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.586921 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.586925 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.586931 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.586935 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.586939 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.586943 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.586948 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.586952 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.586956 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.586960 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.586964 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.586968 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.586972 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.586977 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.586981 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.586985 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.586989 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.586993 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.586997 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.587002 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.587006 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.587010 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.587014 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.587018 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.587022 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.587027 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.587031 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.587035 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.587039 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.587051 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.587055 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.587060 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.587064 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.587068 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.587072 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.587078 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:12 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.587082 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.587086 | controller | 16:12:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.587090 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:13 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.587094 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.587098 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.587103 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.587108 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.587112 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.587116 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.587120 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.587125 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.587129 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.587133 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.587137 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.587141 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.587145 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.587150 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.587154 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.587159 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.587163 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.587168 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.587172 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.587176 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.587180 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.587184 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.587188 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.587192 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.587197 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.587201 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.587205 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.587209 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.587216 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.587220 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.587224 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.587234 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.587239 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.587243 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.587247 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.587251 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.587256 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:13 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.587260 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.587264 | controller | 16:12:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.587268 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:14 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.587272 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.587276 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.587280 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.587284 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.587289 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.587293 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.587298 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.587304 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.587308 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.587312 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.587318 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.587323 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.587327 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.587331 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.587335 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.587339 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.587344 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.587348 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.587352 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.587358 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.587362 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.587366 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.587370 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.587374 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.587378 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.587383 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.587387 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.587391 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.587395 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.587399 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.587403 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.587407 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.587417 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.587422 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.587426 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.587430 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.587434 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:14 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.587438 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.587443 | controller | 16:12:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.587448 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:16 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.587452 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.587457 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.587461 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.587465 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.587469 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.587473 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.587492 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.587497 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.587501 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.587505 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.587509 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.587513 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.587519 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.587523 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.587529 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.587533 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.587537 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.587542 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.587546 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.587551 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.587555 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.587560 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.587564 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.587568 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.587572 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.587576 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.587581 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.587585 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.587589 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.587593 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.587597 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.587601 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.587605 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.587610 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.587620 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.587624 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.587629 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:16 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.587633 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.587637 | controller | 16:12:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.587641 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:17 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.587646 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.587650 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.587654 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.587659 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.587664 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.587668 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.587672 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.587676 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.587680 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.587685 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.587689 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.587703 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.587709 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.587715 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.587719 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.587723 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.587727 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.587731 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.587736 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.587740 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.587744 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.587748 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.587752 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.587756 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.587761 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.587765 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.587769 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.587773 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.587777 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.587781 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.587786 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.587790 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.587794 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.587798 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.587802 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.587813 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.587819 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:17 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.587823 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.587828 | controller | 16:12:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.587832 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:18 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.587836 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.587840 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.587844 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.587849 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.587853 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.587857 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.587861 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.587865 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.587869 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.587873 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.587878 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.587882 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.587886 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.587890 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.587894 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.587898 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.587902 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.587907 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.587911 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.587915 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.587919 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.587923 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.587927 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.587931 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.587936 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.587940 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.587944 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.587948 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.587953 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.587959 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.587963 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.587967 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.587972 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.587976 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.587980 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.587984 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.587988 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:18 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.588000 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.588004 | controller | 16:12:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.588009 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:19 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.588013 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.588017 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.588021 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.588025 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.588029 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.588033 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.588038 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.588042 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.588046 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.588050 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.588054 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.588058 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.588063 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.588067 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.588071 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.588075 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.588079 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.588083 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.588087 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.588093 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.588097 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.588102 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.588106 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.588110 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.588114 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.588118 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.588122 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.588127 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.588131 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.588135 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.588139 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.588143 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.588147 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.588152 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.588156 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.588160 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.588164 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:19 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.588168 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.588178 | controller | 16:12:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.588182 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:20 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.588187 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.588191 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.588195 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.588199 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.588203 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.588208 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.588212 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.588216 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.588220 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.588224 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.588228 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.588232 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.588240 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.588244 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.588248 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.588252 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.588256 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.588260 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.588264 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.588269 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.588273 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.588277 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.588281 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.588285 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.588289 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.588293 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.588298 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.588302 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.588306 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.588310 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.588314 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.588318 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.588323 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.588327 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.588331 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.588335 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.588339 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:20 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.588343 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.588348 | controller | 16:12:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.588358 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:21 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.588362 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.588366 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.588370 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.588379 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.588383 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.588387 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.588391 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.588395 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.588400 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.588404 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.588408 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.588412 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.588416 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.588420 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.588425 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.588429 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.588433 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.588437 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.588441 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.588447 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.588452 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.588456 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.588460 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.588464 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.588468 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.588472 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.588477 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.588481 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.588485 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.588489 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.588493 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.588497 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.588502 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.588506 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.588511 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.588516 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.588521 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:21 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.588525 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.588529 | controller | 16:12:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.588534 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:23 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.588538 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.588548 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.588552 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.588557 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.588561 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.588565 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.588569 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.588574 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.588578 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.588582 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.588586 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.588590 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.588595 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.588599 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.588603 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.588607 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.588611 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.588616 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.588620 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.588624 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.588628 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.588633 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.588637 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.588641 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.588645 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.588650 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.588654 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.588658 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.588664 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.588668 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.588672 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.588676 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.588681 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.588685 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.588689 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.588706 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.588710 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:23 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.588716 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.588721 | controller | 16:12:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.588725 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:24 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.588729 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.588733 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.588744 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.588749 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.588753 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.588757 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.588761 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.588766 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.588770 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.588774 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.588778 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.588784 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.588788 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.588792 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.588798 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.588802 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.588807 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.588811 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.588815 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.588821 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.588825 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.588829 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.588834 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.588838 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.588842 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.588846 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.588851 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.588855 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.588859 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.588863 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.588868 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.588872 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.588876 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.588880 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.588885 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.588889 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.588893 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:24 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.588897 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.588902 | controller | 16:12:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.588906 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:25 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.588910 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.588914 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.588918 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.588928 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.588933 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.588937 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.588941 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.588945 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.588950 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.588954 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.588958 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.588962 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.588968 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.588972 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.588976 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.588980 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.588985 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.588989 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.588993 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.588998 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.589002 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.589008 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.589013 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.589019 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.589025 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.589030 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.589036 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.589041 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.589046 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.589050 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.589054 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.589060 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.589065 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.589069 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.589073 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.589077 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.589082 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:25 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.589086 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.589090 | controller | 16:12:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.589094 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:26 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.589099 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.589103 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.589107 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.589114 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.589118 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.589129 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.589134 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.589139 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.589143 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.589147 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.589151 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.589155 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.589160 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.589164 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.589168 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.589173 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.589179 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.589185 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.589191 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.589197 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.589203 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.589209 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.589214 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.589219 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.589223 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.589227 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.589231 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.589235 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.589240 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.589246 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.589251 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.589255 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.589259 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.589264 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.589268 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.589272 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.589278 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:26 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.589282 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.589286 | controller | 16:12:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.589290 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:27 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.589295 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.589299 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.589303 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.589307 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.589312 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.589316 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.589327 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.589331 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.589335 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.589340 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.589347 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.589351 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.589356 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.589360 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.589364 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.589368 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.589373 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.589378 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.589384 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.589392 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.589398 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.589404 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.589409 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.589415 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.589421 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.589427 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.589432 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.589438 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.589445 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.589449 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.589453 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.589457 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.589462 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.589466 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.589470 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.589474 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.589479 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:27 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.589483 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.589487 | controller | 16:12:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.589491 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:28 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.589496 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.589500 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.589504 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.589508 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.589513 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.589517 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.589521 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.589525 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.589536 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.589541 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.589545 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.589550 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.589554 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.589558 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.589562 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.589566 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.589571 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.589575 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.589579 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.589585 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.589589 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.589593 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.589598 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.589602 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.589606 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.589610 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.589614 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.589621 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.589627 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.589632 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.589638 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.589644 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.589649 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.589654 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.589659 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.589663 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.589667 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:28 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.589671 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.589676 | controller | 16:12:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.589680 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:29 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.589686 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.589705 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.589713 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.589720 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.589724 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.589729 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.589733 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.589737 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.589741 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.589753 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.589758 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.589762 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.589768 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.589773 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.589779 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.589784 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.589790 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.589796 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.589802 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.589808 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.589813 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.589819 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.589824 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.589830 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.589835 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.589840 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.589845 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.589851 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.589857 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.589863 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.589869 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.589873 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.589877 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.589882 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.589886 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.589890 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.589894 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:29 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.589899 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.589903 | controller | 16:12:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.589907 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:31 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.589911 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.589915 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.589920 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.589926 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.589930 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.589935 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.589939 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.589943 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.589947 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.589954 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.589958 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.589970 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.589975 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.589980 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.589984 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.589988 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.589992 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.589998 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.590003 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.590009 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.590015 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.590020 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.590025 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.590029 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.590033 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.590037 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.590042 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.590046 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.590051 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.590057 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.590062 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.590068 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.590073 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.590077 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.590081 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.590085 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.590092 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:31 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.590096 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.590100 | controller | 16:12:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.590105 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:32 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.590109 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.590113 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.590117 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.590121 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.590126 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.590130 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.590135 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.590141 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.590146 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.590152 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.590158 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.590164 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.590178 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.590182 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.590190 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.590194 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.590199 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.590203 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.590207 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.590212 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.590216 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.590220 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.590224 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.590228 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.590233 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.590237 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.590243 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.590247 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.590253 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.590257 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.590262 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.590266 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.590270 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.590274 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.590279 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.590283 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.590287 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:32 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.590291 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.590295 | controller | 16:12:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.590300 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:33 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.590304 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.590308 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.590312 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.590317 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.590321 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.590325 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.590329 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.590333 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.590338 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.590342 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.590346 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.590350 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.590355 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.590367 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.590371 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.590376 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.590380 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.590384 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.590389 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.590396 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.590400 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.590405 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.590409 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.590413 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.590417 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.590421 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.590426 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.590430 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.590434 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.590438 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.590442 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.590447 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.590451 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.590455 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.590459 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.590464 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.590468 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:33 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.590472 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.590476 | controller | 16:12:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.590481 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:34 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.590485 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.590489 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.590493 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.590497 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.590502 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.590506 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.590510 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.590514 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.590521 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.590526 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.590530 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.590534 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.590540 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.590544 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.590548 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.590560 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.590565 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.590569 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.590574 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.590578 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.590582 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.590586 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.590591 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.590595 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.590599 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.590603 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.590607 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.590612 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.590616 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.590622 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.590626 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.590630 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.590635 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.590639 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.590643 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.590647 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.590652 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:34 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.590656 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.590660 | controller | 16:12:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.590664 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:35 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.590669 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.590673 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.590677 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.590683 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.590687 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.590704 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.590711 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.590716 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.590720 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.590724 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.590728 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.590732 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.590737 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.590741 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.590745 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.590749 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.590760 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.590765 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.590769 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.590773 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.590778 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.590782 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.590786 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.590790 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.590796 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.590801 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.590805 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.590809 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.590813 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.590818 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.590822 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.590826 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.590830 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.590834 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.590839 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.590843 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.590848 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:35 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.590853 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.590857 | controller | 16:12:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.590861 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:36 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.590865 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.590870 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.590874 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.590878 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.590884 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.590888 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.590892 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.590897 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.590901 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.590905 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.590909 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.590914 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.590918 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.590922 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.590926 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.590930 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.590935 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.590946 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.590951 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.590955 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.590959 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.590964 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.590968 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.590972 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.590976 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.590981 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.590985 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.590989 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.590996 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.591001 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.591005 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.591009 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.591014 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.591018 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.591022 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.591026 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.591030 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:37 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.591035 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.591039 | controller | 16:12:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.591043 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:38 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.591047 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.591052 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.591056 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.591062 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.591066 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.591070 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.591074 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.591080 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.591084 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.591089 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.591093 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.591097 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.591101 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.591105 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.591110 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.591114 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.591118 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.591122 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.591133 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.591139 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.591144 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.591148 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.591152 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.591157 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.591161 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.591165 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.591169 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.591173 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.591177 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.591182 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.591186 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.591190 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.591194 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.591199 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.591203 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.591207 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.591211 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:38 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.591215 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.591220 | controller | 16:12:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.591224 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:39 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.591228 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.591232 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.591236 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.591241 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.591245 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.591249 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.591253 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.591258 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.591262 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.591266 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.591270 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.591274 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.591280 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.591284 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.591288 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.591293 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.591297 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.591301 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.591305 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.591311 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.591322 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.591326 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.591331 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.591337 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.591341 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.591345 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.591349 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.591353 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.591358 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.591362 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.591366 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.591370 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.591374 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.591379 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.591383 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.591387 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.591391 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:39 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.591396 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.591400 | controller | 16:12:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.591404 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:40 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.591408 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.591413 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.591417 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.591424 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.591428 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.591432 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.591437 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.591441 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.591445 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.591449 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.591454 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.591458 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.591462 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.591466 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.591472 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.591476 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.591505 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.591510 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.591514 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.591518 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.591523 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.591534 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.591538 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.591543 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.591547 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.591551 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.591555 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.591559 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.591564 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.591568 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.591572 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.591576 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.591580 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.591585 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.591589 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.591593 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.591599 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:40 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.591603 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.591608 | controller | 16:12:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.591612 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:41 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.591616 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.591620 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.591626 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.591631 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.591635 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.591639 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.591643 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.591648 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.591652 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.591656 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.591660 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.591664 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.591669 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.591673 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.591677 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.591681 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.591685 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.591690 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.591706 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.591713 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.591717 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.591722 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.591726 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.591737 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.591742 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.591746 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.591750 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.591754 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.591761 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.591765 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.591769 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.591774 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.591778 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.591782 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.591786 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.591790 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.591795 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:41 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.591799 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.591803 | controller | 16:12:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.591807 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:42 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.591811 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.591816 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.591820 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.591824 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.591828 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.591832 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.591837 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.591841 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.591845 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.591849 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.591853 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.591857 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.591862 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.591866 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.591872 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.591876 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.591880 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.591884 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.591889 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.591894 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.591898 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.591903 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.591908 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.591913 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.591924 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.591928 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.591933 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.591937 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.591941 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.591947 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.591951 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.591955 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.591960 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.591964 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.591968 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.591972 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.591976 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:42 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.591981 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.591985 | controller | 16:12:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.591989 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:44 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.591993 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.591998 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.592002 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.592006 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.592010 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.592014 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.592019 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.592023 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.592027 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.592031 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.592035 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.592039 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.592045 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.592049 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.592054 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.592058 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.592062 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.592066 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.592071 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.592075 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.592079 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.592083 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.592087 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.592092 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.592096 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.592100 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.592111 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.592116 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.592120 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.592124 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.592128 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.592133 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.592137 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.592141 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.592145 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.592149 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.592154 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:44 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.592158 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.592162 | controller | 16:12:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.592166 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:45 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.592171 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.592181 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.592185 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.592191 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.592195 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.592199 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.592204 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.592208 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.592212 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.592216 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.592220 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.592225 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.592229 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.592233 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.592237 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.592241 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.592246 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.592250 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.592254 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.592260 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.592264 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.592268 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.592273 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.592277 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.592281 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.592285 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.592289 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.592300 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.592305 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.592309 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.592314 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.592318 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.592322 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.592326 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.592331 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.592335 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.592340 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:45 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.592345 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.592349 | controller | 16:12:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.592353 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:46 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.592357 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.592361 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.592366 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.592371 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.592376 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.592380 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.592384 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.592388 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.592392 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.592396 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.592401 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.592405 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.592409 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.592413 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.592417 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.592421 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.592425 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.592430 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.592434 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.592438 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.592445 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.592449 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.592453 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.592458 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.592462 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.592466 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.592470 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.592474 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.592487 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.592491 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.592495 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.592500 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.592504 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.592508 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.592512 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.592516 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.592520 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:46 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.592524 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.592529 | controller | 16:12:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.592533 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:47 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.592537 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.592541 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.592545 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.592549 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.592554 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.592558 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.592562 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.592566 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.592570 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.592575 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.592579 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.592583 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.592587 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.592591 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.592595 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.592600 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.592604 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.592608 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.592612 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.592618 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.592622 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.592626 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.592630 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.592635 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.592639 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.592643 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.592647 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.592651 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.592655 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.592660 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.592670 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.592674 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.592679 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.592683 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.592687 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.592706 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.592712 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:47 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.592717 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.592721 | controller | 16:12:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.592725 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:48 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.592731 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.592736 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.592740 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.592744 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.592748 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.592752 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.592757 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.592761 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.592765 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.592769 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.592774 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.592778 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.592784 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.592788 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.592793 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.592798 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.592802 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.592806 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.592810 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.592815 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.592819 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.592823 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.592827 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.592832 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.592836 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.592840 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.592844 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.592848 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.592852 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.592857 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.592861 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.592871 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.592876 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.592880 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.592884 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.592889 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.592893 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:48 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.592897 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.592901 | controller | 16:12:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.592905 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:50 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.592910 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.592914 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.592918 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.592924 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.592928 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.592932 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.592936 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.592941 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.592945 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.592949 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.592953 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.592958 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.592962 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.592966 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.592970 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.592976 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.592981 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.592985 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.592989 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.592993 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.592998 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.593002 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.593006 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.593010 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.593015 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.593019 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.593023 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.593027 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.593031 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.593036 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.593040 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.593044 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.593054 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.593059 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.593063 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.593067 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.593073 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:50 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.593079 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.593083 | controller | 16:12:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.593088 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:51 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.593092 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.593096 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.593100 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.593104 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.593109 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.593113 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.593117 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.593121 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.593125 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.593130 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.593134 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.593138 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.593142 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.593146 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.593151 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.593155 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.593159 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.593163 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.593168 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.593173 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.593177 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.593182 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.593186 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.593190 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.593194 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.593199 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.593203 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.593207 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.593213 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.593218 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.593223 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.593227 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.593231 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.593235 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.593246 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.593250 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.593255 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:51 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.593259 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.593263 | controller | 16:12:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.593269 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:53 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.593273 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.593278 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.593282 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.593286 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.593290 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.593295 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.593300 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.593305 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.593310 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.593314 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.593320 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.593325 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.593329 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.593333 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.593337 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.593341 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.593346 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.593350 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.593354 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.593360 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.593364 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.593368 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.593372 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.593377 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.593381 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.593385 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.593389 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.593394 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.593398 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.593402 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.593406 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.593410 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.593414 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.593419 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.593423 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.593433 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.593438 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:53 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.593442 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.593446 | controller | 16:12:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.593451 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:54 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.593455 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.593459 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.593463 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.593467 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.593472 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.593476 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.593480 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.593484 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.593488 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.593493 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.593497 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.593501 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.593508 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.593513 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.593518 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.593522 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.593526 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.593530 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.593534 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.593539 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.593543 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.593547 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.593551 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.593556 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.593560 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.593564 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.593568 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.593572 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.593577 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.593581 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.593585 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.593589 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.593593 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.593598 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.593602 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.593606 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.593618 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:54 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.593623 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.593627 | controller | 16:12:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.593632 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:55 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.593636 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.593640 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.593644 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.593650 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.593654 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.593658 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.593663 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.593667 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.593671 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.593675 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.593680 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.593684 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.593688 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.593703 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.593710 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.593714 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.593718 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.593722 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.593727 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.593731 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.593735 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.593739 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.593744 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.593748 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.593752 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.593756 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.593761 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.593765 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.593769 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.593773 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.593777 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.593782 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.593786 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.593790 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.593794 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.593799 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.593805 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:55 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.593811 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.593821 | controller | 16:12:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.593826 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:56 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.593830 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.593834 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.593839 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.593843 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.593847 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.593851 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.593856 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.593860 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.593864 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.593868 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.593872 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.593877 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.593881 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.593885 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.593889 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.593894 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.593898 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.593902 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.593906 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.593910 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.593914 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.593919 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.593923 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.593927 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.593931 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.593935 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.593940 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.593944 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.593949 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.593954 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.593958 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.593962 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.593966 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.593971 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.593975 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.593979 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.593983 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:56 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.593987 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.593992 | controller | 16:12:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.594002 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:57 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.594007 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.594011 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.594015 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.594019 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.594024 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.594028 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.594032 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.594036 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.594040 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.594045 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.594049 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.594053 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.594059 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.594063 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.594068 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.594073 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.594077 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.594081 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.594086 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.594092 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.594097 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.594101 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.594105 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.594110 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.594114 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.594118 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.594122 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.594126 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.594131 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.594135 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.594139 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.594143 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.594148 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.594152 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.594156 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.594160 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.594165 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:57 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.594169 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.594173 | controller | 16:12:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.594177 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:58 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.594188 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.594193 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.594197 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.594202 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.594206 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.594210 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.594216 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.594220 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.594224 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.594228 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.594233 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.594237 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.594242 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.594247 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.594251 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.594255 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.594259 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.594264 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.594268 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.594272 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.594276 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.594281 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.594285 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.594289 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.594293 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.594297 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.594302 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.594306 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.594310 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.594314 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.594319 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.594323 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.594327 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.594331 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.594337 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.594341 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.594346 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:58 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.594350 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.594354 | controller | 16:12:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.594358 | controller | \"ac-watcher\" not found\n logger.go:42: 16:12:59 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.594362 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.594367 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.594377 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.594383 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.594387 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.594392 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.594396 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.594400 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.594404 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.594408 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.594412 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.594417 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.594421 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.594425 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.594429 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.594433 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.594438 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.594442 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.594446 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.594450 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.594455 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.594459 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.594463 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.594467 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.594471 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.594475 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.594480 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.594484 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.594488 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.594495 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.594500 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.594504 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.594508 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.594512 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.594516 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.594521 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.594526 | controller | expected application_credential_id\"]\n logger.go:42: 16:12:59 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.594531 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.594535 | controller | 16:13:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.594539 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:01 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.594543 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.594548 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.594552 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.594562 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.594566 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.594570 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.594575 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.594579 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.594583 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.594587 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.594592 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.594598 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.594602 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.594606 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.594610 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.594615 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.594619 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.594623 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.594627 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.594631 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.594636 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.594640 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.594644 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.594648 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.594652 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.594657 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.594661 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.594665 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.594671 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.594675 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.594679 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.594683 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.594687 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.594709 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.594717 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.594721 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.594726 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:01 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.594730 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.594734 | controller | 16:13:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.594739 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:02 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.594743 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.594747 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.594751 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.594755 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.594760 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.594770 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.594775 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.594779 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.594783 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.594788 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.594792 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.594796 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.594800 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.594804 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.594809 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.594813 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.594817 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.594821 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.594825 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.594831 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.594836 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.594840 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.594844 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.594848 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.594852 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.594857 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.594861 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.594865 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.594869 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.594873 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.594877 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.594884 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.594888 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.594892 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.594896 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.594901 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.594905 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:02 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.594909 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.594913 | controller | 16:13:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.594917 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:03 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.594922 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.594926 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.594930 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.594936 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.594940 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.594944 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.594954 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.594959 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.594963 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.594967 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.594972 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.594976 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.594982 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.594986 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.594990 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.594995 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.594999 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.595003 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.595007 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.595013 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.595017 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.595021 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.595025 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.595030 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.595034 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.595038 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.595042 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.595046 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.595051 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.595055 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.595059 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.595063 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.595068 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.595072 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.595076 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.595080 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.595085 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:03 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.595089 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.595093 | controller | 16:13:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.595097 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:04 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.595102 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.595106 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.595110 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.595116 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.595122 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.595126 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.595130 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.595141 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.595146 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.595150 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.595154 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.595159 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.595163 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.595167 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.595171 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.595177 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.595181 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.595186 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.595190 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.595194 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.595198 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.595203 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.595207 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.595211 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.595215 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.595219 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.595224 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.595228 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.595232 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.595236 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.595240 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.595245 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.595249 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.595253 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.595257 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.595262 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.595267 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:04 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.595272 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.595276 | controller | 16:13:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.595280 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:05 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.595284 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.595289 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.595293 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.595297 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.595301 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.595305 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.595310 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.595314 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.595318 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.595328 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.595333 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.595337 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.595341 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.595346 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.595351 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.595356 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.595360 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.595364 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.595368 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.595373 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.595377 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.595381 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.595385 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.595390 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.595394 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.595398 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.595402 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.595408 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.595415 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.595419 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.595423 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.595427 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.595432 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.595436 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.595440 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.595444 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.595448 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:05 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.595453 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.595457 | controller | 16:13:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.595461 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:06 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.595466 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.595470 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.595474 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.595478 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.595504 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.595509 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.595513 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.595518 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.595522 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.595526 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.595537 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.595542 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.595546 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.595550 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.595555 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.595559 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.595563 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.595567 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.595571 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.595577 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.595581 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.595586 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.595590 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.595594 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.595598 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.595603 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.595607 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.595611 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.595615 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.595619 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.595624 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.595628 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.595632 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.595636 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.595640 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.595645 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.595649 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:06 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.595653 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.595657 | controller | 16:13:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.595661 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:08 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.595666 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.595670 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.595674 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.595678 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.595683 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.595687 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.595691 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.595709 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.595713 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.595720 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.595724 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.595728 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.595742 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.595747 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.595751 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.595755 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.595759 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.595764 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.595768 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.595772 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.595776 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.595781 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.595785 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.595789 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.595793 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.595797 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.595801 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.595806 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.595810 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.595816 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.595820 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.595824 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.595829 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.595833 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.595837 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.595841 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.595846 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:08 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.595850 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.595854 | controller | 16:13:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.595858 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:09 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.595863 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.595867 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.595871 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.595877 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.595881 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.595885 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.595889 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.595894 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.595898 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.595902 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.595906 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.595910 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.595915 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.595925 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.595930 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.595934 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.595939 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.595943 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.595947 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.595953 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.595957 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.595961 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.595966 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.595970 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.595974 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.595978 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.595984 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.595988 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.595992 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.595997 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.596001 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.596005 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.596009 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.596013 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.596018 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.596022 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.596028 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:09 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.596032 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.596036 | controller | 16:13:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.596040 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:10 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.596045 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.596049 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.596053 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.596057 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.596061 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.596066 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.596070 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.596074 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.596078 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.596082 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.596086 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.596091 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.596095 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.596099 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.596103 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.596115 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.596119 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.596123 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.596128 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.596132 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.596136 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.596140 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.596145 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.596149 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.596153 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.596157 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.596161 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.596166 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.596171 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.596176 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.596180 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.596184 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.596188 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.596193 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.596197 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.596201 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.596205 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:10 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.596209 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.596213 | controller | 16:13:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.596218 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:11 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.596222 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.596226 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.596230 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.596236 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.596240 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.596245 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.596249 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.596253 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.596259 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.596263 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.596267 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.596272 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.596276 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.596280 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.596284 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.596288 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.596301 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.596306 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.596310 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.596318 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.596323 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.596327 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.596331 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.596335 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.596340 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.596344 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.596348 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.596352 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.596356 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.596361 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.596365 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.596369 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.596373 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.596378 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.596382 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.596386 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.596390 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:11 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.596395 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.596399 | controller | 16:13:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.596403 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:12 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.596408 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.596412 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.596416 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.596420 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.596425 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.596429 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.596433 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.596437 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.596441 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.596446 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.596450 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.596454 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.596460 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.596464 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.596468 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.596472 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.596477 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.596488 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.596492 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.596497 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.596501 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.596505 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.596510 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.596514 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.596520 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.596524 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.596528 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.596532 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.596548 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.596553 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.596557 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.596561 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.596565 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.596570 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.596574 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.596578 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.596582 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:12 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.596586 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.596591 | controller | 16:13:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.596595 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:13 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.596599 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.596603 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.596608 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.596615 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.596619 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.596623 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.596627 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.596631 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.596636 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.596640 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.596644 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.596648 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.596653 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.596657 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.596663 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.596667 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.596671 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.596675 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.596686 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.596691 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.596708 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.596713 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.596717 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.596721 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.596725 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.596730 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.596734 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.596738 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.596742 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.596747 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.596751 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.596755 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.596759 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.596764 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.596768 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.596772 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.596778 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:13 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.596782 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.596786 | controller | 16:13:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.596791 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:14 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.596795 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.596799 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.596803 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.596807 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.596812 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.596816 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.596820 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.596826 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.596830 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.596835 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.596839 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.596843 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.596847 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.596851 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.596856 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.596860 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.596864 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.596868 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.596873 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.596879 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.596890 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.596894 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.596899 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.596903 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.596907 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.596911 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.596915 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.596920 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.596925 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.596930 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.596934 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.596938 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.596942 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.596947 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.596951 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.596955 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.596959 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:14 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.596964 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.596968 | controller | 16:13:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.596972 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:16 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.596976 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.596981 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.596985 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.596989 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.596993 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.596997 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.597002 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.597006 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.597010 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.597014 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.597019 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.597023 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.597027 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.597031 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.597036 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.597040 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.597044 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.597048 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.597052 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.597058 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.597062 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.597072 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.597077 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.597083 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.597087 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.597091 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.597096 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.597100 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.597104 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.597110 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.597114 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.597118 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.597123 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.597127 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.597131 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.597135 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.597139 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:16 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.597144 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.597148 | controller | 16:13:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.597152 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:17 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.597156 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.597161 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.597165 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.597169 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.597175 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.597179 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.597183 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.597188 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.597192 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.597196 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.597200 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.597205 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.597210 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.597214 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.597218 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.597223 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.597227 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.597231 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.597236 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.597240 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.597244 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.597248 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.597253 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.597263 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.597267 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.597272 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.597276 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.597280 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.597284 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.597288 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.597293 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.597297 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.597301 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.597305 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.597310 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.597314 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.597318 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:17 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.597324 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.597328 | controller | 16:13:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.597333 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:18 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.597337 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.597341 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.597347 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.597352 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.597357 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.597361 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.597365 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.597369 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.597374 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.597378 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.597382 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.597386 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.597391 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.597395 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.597399 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.597403 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.597407 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.597412 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.597416 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.597420 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.597424 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.597429 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.597433 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.597437 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.597447 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.597452 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.597456 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.597460 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.597465 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.597469 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.597473 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.597477 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.597482 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.597486 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.597490 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.597494 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.597500 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:18 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.597504 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.597508 | controller | 16:13:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.597513 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:19 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.597517 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.597521 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.597525 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.597532 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.597537 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.597541 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.597545 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.597549 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.597554 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.597558 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.597562 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.597566 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.597571 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.597575 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.597579 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.597583 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.597587 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.597592 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.597596 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.597600 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.597604 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.597609 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.597614 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.597619 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.597623 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.597627 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.597638 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.597642 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.597648 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.597653 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.597657 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.597661 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.597666 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.597670 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.597674 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.597678 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.597682 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:19 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.597687 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.597691 | controller | 16:13:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.597711 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:20 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.597715 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.597719 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.597724 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.597728 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.597732 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.597736 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.597740 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.597745 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.597749 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.597753 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.597757 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.597762 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.597766 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.597770 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.597774 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.597778 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.597783 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.597787 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.597791 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.597798 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.597803 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.597807 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.597811 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.597815 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.597820 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.597824 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.597828 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.597839 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.597843 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.597848 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.597852 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.597856 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.597860 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.597865 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.597869 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.597873 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.597877 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:20 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.597881 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.597886 | controller | 16:13:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.597890 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:21 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.597894 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.597900 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.597904 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.597908 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.597913 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.597917 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.597921 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.597925 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.597929 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.597934 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.597938 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.597942 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.597948 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.597952 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.597958 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.597962 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.597966 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.597970 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.597975 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.597979 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.597983 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.597987 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.597991 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.597996 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.598000 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.598004 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.598008 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.598013 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.598023 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.598027 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.598032 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.598036 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.598040 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.598044 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.598049 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.598053 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.598057 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:22 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.598061 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.598065 | controller | 16:13:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.598070 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:23 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.598074 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.598078 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.598082 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.598088 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.598092 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.598097 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.598101 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.598105 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.598109 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.598114 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.598118 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.598122 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.598139 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.598144 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.598148 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.598152 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.598156 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.598161 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.598165 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.598169 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.598175 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.598179 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.598183 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.598188 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.598192 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.598196 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.598200 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.598204 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.598209 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.598213 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.598223 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.598228 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.598232 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.598236 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.598241 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.598245 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.598251 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:23 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.598255 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.598259 | controller | 16:13:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.598263 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:24 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.598268 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.598272 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.598276 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.598280 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.598284 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.598289 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.598293 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.598297 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.598301 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.598305 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.598310 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.598314 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.598318 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.598322 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.598326 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.598331 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.598335 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.598339 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.598343 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.598348 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.598352 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.598356 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.598360 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.598365 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.598369 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.598373 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.598377 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.598381 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.598387 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.598393 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.598397 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.598407 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.598412 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.598416 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.598421 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.598425 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.598429 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:24 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.598433 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.598438 | controller | 16:13:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.598442 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:25 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.598448 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.598452 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.598456 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.598460 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.598465 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.598469 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.598473 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.598477 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.598481 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.598486 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.598490 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.598494 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.598498 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.598502 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.598507 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.598511 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.598515 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.598519 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.598524 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.598529 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.598534 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.598538 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.598542 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.598546 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.598550 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.598555 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.598559 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.598563 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.598567 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.598571 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.598576 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.598580 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.598590 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.598594 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.598599 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.598603 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.598607 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:25 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.598612 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.598616 | controller | 16:13:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.598620 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:26 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.598624 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.598628 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.598632 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.598637 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.598641 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.598645 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.598649 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.598653 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.598658 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.598662 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.598666 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.598670 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.598679 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.598683 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.598687 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.598703 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.598710 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.598714 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.598718 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.598724 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.598728 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.598733 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.598737 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.598741 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.598745 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.598749 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.598754 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.598758 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.598762 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.598766 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.598770 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.598775 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.598779 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.598789 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.598794 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.598798 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.598803 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:26 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.598807 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.598811 | controller | 16:13:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.598815 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:27 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.598819 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.598824 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.598828 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.598835 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.598840 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.598844 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.598848 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.598852 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.598857 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.598861 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.598865 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.598869 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.598873 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.598878 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.598882 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.598886 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.598890 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.598894 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.598898 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.598903 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.598907 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.598911 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.598915 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.598919 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.598924 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.598928 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.598932 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.598936 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.598940 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.598945 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.598950 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.598955 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.598959 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.598963 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.598967 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.598978 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.598984 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:27 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.598998 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.599002 | controller | 16:13:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.599008 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:28 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.599012 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.599017 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.599021 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.599025 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.599029 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.599033 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.599038 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.599042 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.599046 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.599050 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.599054 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.599059 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.599063 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.599067 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.599071 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.599075 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.599087 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.599092 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.599096 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.599107 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.599112 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.599116 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.599120 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.599124 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.599129 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.599133 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.599137 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.599141 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.599147 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.599151 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.599155 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.599160 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.599172 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.599176 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.599180 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.599185 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.599194 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:28 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.599199 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.599203 | controller | 16:13:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.599208 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:30 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.599212 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.599216 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.599220 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.599225 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.599229 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.599233 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.599237 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.599241 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.599246 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.599250 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.599254 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.599258 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.599263 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.599268 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.599274 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.599278 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.599282 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.599287 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.599291 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.599296 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.599301 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.599305 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.599309 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.599313 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.599318 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.599322 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.599326 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.599330 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.599334 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.599339 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.599343 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.599347 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.599351 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.599355 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.599360 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.599364 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.599368 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:30 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.599372 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.599383 | controller | 16:13:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.599387 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:31 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.599392 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.599396 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.599400 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.599405 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.599410 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.599415 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.599419 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.599423 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.599427 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.599432 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.599436 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.599440 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.599446 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.599450 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.599454 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.599458 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.599463 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.599467 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.599471 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.599475 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.599480 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.599484 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.599499 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.599503 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.599508 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.599512 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.599516 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.599520 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.599524 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.599529 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.599533 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.599537 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.599541 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.599546 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.599550 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.599554 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.599558 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:31 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.599564 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.599569 | controller | 16:13:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.599579 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:32 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.599584 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.599588 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.599592 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.599599 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.599604 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.599608 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.599612 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.599616 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.599620 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.599625 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.599629 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.599633 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.599637 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.599649 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.599654 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.599658 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.599662 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.599666 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.599670 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.599676 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.599681 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.599685 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.599689 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.599704 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.599710 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.599715 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.599719 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.599723 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.599727 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.599733 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.599738 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.599742 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.599746 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.599751 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.599755 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.599759 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.599765 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:32 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.599769 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.599773 | controller | 16:13:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.599778 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:33 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.599789 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.599793 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.599798 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.599802 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.599806 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.599810 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.599815 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.599819 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.599823 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.599827 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.599831 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.599836 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.599841 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.599846 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.599850 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.599854 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.599858 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.599862 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.599867 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.599871 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.599875 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.599879 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.599884 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.599888 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.599892 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.599896 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.599901 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.599905 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.599910 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.599915 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.599919 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.599923 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.599927 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.599931 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.599936 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.599940 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.599944 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:33 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.599948 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.599953 | controller | 16:13:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.599957 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:34 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.599961 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.599965 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.599976 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.599981 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.599985 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.599989 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.599993 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.599998 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.600002 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.600006 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.600010 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.600014 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.600019 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.600023 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.600027 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.600031 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.600036 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.600040 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.600044 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.600050 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.600054 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.600058 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.600062 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.600066 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.600070 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.600075 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.600079 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.600083 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.600087 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.600091 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.600095 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.600099 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.600104 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.600108 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.600114 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.600118 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.600122 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:34 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.600126 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.600130 | controller | 16:13:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.600134 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:35 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.600139 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.600143 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.600147 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.600158 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.600163 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.600167 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.600171 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.600176 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.600180 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.600184 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.600188 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.600192 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.600198 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.600202 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.600206 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.600210 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.600215 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.600219 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.600223 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.600227 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.600231 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.600236 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.600240 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.600244 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.600248 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.600252 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.600256 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.600261 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.600265 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.600269 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.600273 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.600277 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.600281 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.600286 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.600290 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.600294 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.600298 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:35 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.600302 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.600307 | controller | 16:13:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.600311 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:36 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.600315 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.600319 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.600324 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.600329 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.600339 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.600343 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.600348 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.600352 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.600356 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.600360 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.600364 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.600370 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.600374 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.600378 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.600383 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.600387 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.600391 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.600395 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.600399 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.600404 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.600408 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.600412 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.600416 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.600420 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.600424 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.600428 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.600433 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.600437 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.600441 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.600445 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.600449 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.600453 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.600457 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.600462 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.600466 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.600470 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.600475 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:36 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.600480 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.600484 | controller | 16:13:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.600488 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:38 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.600492 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.600497 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.600501 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.600505 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.600509 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.600513 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.600523 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.600527 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.600531 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.600536 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.600540 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.600544 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.600548 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.600552 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.600558 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.600562 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.600566 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.600570 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.600575 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.600591 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.600595 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.600599 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.600603 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.600608 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.600612 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.600616 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.600620 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.600624 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.600631 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.600636 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.600640 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.600645 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.600650 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.600654 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.600658 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.600662 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.600666 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:38 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.600671 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.600675 | controller | 16:13:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.600679 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:39 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.600683 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.600687 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.600707 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.600714 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.600718 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.600722 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.600726 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.600737 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.600741 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.600746 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.600750 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.600754 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.600758 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.600762 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.600766 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.600770 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.600775 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.600779 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.600783 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.600789 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.600793 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.600797 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.600802 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.600806 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.600810 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.600814 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.600818 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.600822 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.600827 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.600831 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.600835 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.600839 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.600843 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.600847 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.600851 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.600856 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.600860 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:39 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.600864 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.600868 | controller | 16:13:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.600872 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:40 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.600885 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.600889 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.600893 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.600897 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.600901 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.600906 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.600910 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.600914 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.600918 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.600928 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.600935 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.600939 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.600945 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.600949 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.600953 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.600957 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.600961 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.600965 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.600970 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.600974 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.600978 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.600982 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.600986 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.600990 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.600995 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.600999 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.601003 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.601007 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.601011 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.601017 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.601021 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.601025 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.601029 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.601033 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.601037 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.601042 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.601046 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:40 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.601050 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.601054 | controller | 16:13:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.601058 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:41 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.601071 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.601075 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.601080 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.601085 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.601089 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.601094 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.601098 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.601102 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.601106 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.601110 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.601120 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.601125 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.601129 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.601133 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.601137 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.601141 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.601146 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.601150 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.601154 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.601158 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.601170 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.601174 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.601178 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.601182 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.601186 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.601190 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.601195 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.601200 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.601205 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.601209 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.601213 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.601217 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.601221 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.601226 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.601230 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.601234 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.601239 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:41 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.601244 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.601248 | controller | 16:13:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.601252 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:42 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.601256 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.601260 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.601264 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.601275 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.601280 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.601284 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.601288 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.601292 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.601296 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.601301 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.601305 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.601309 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.601319 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.601323 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.601327 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.601331 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.601336 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.601340 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.601344 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.601348 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.601352 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.601356 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.601362 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.601366 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.601371 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.601375 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.601379 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.601383 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.601389 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.601393 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.601397 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.601401 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.601406 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.601410 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.601414 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.601418 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.601422 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:42 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.601502 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.601507 | controller | 16:13:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.601512 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:43 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.601516 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.601520 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.601524 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.601530 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.601534 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.601539 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.601543 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.601547 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.601551 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.601557 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.601561 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.601565 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.601569 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.601580 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.601585 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.601597 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.601601 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.601605 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.601610 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.601617 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.601622 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.601626 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.601630 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.601634 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.601638 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.601642 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.601646 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.601650 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.601654 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.601659 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.601663 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.601667 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.601671 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.601675 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.601679 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.601684 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.601688 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:43 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.601711 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.601723 | controller | 16:13:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.601727 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:45 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.601731 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.601735 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.601739 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.601743 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.601748 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.601752 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.601756 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.601760 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.601764 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.601768 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.601779 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.601784 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.601790 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.601794 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.601798 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.601810 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.601814 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.601819 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.601823 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.601827 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.601831 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.601835 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.601840 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.601844 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.601848 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.601852 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.601858 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.601862 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.601867 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.601871 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.601875 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.601879 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.601883 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.601888 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.601892 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.601896 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.601900 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:45 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.601904 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.601908 | controller | 16:13:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.601912 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:46 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.601917 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.601921 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.601925 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.601931 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.601935 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.601939 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.601943 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.601947 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.601952 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.601956 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.601985 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.601990 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.601994 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.601998 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.602004 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.602009 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.602020 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.602024 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.602028 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.602033 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.602037 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.602041 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.602045 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.602049 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.602053 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.602058 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.602062 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.602066 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.602070 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.602074 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.602079 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.602083 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.602087 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.602091 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.602095 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.602099 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.602105 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:46 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.602109 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.602114 | controller | 16:13:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.602118 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:47 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.602122 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.602126 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.602130 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.602134 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.602139 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.602143 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.602147 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.602151 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.602157 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.602161 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.602166 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.602170 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.602174 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.602178 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.602182 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.602186 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.602191 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.602202 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.602206 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.602211 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.602215 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.602219 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.602223 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.602227 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.602231 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.602236 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.602240 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.602244 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.602250 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.602254 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.602258 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.602262 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.602266 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.602271 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.602275 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.602279 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.602283 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:47 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.602287 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.602292 | controller | 16:13:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.602297 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:48 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.602302 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.602307 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.602312 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.602317 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.602322 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.602326 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.602330 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.602334 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.602338 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.602343 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.602347 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.602351 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.602355 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.602359 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.602363 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.602368 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.602372 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.602376 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.602386 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.602392 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.602396 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.602401 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.602405 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.602409 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.602415 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.602419 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.602423 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.602427 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.602432 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.602437 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.602442 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.602446 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.602450 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.602454 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.602458 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.602463 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.602467 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:48 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.602471 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.602475 | controller | 16:13:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.602479 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:49 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.602483 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.602488 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.602492 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.602496 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.602500 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.602504 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.602508 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.602512 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.602517 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.602521 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.602525 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.602529 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.602535 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.602539 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.602543 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.602547 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.602551 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.602556 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.602560 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.602572 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.602577 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.602581 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.602585 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.602590 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.602594 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.602598 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.602602 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.602606 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.602610 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.602615 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.602619 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.602623 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.602627 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.602631 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.602636 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.602640 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.602644 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:49 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.602648 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.602652 | controller | 16:13:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.602657 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:50 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.602661 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.602665 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.602669 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.602675 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.602679 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.602683 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.602687 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.602704 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.602711 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.602715 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.602719 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.602724 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.602728 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.602732 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.602736 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.602740 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.602744 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.602748 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.602753 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.602757 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.602761 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.602772 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.602777 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.602781 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.602785 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.602789 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.602794 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.602798 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.602802 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.602806 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.602810 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.602814 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.602819 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.602823 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.602827 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.602831 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.602838 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:50 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.602842 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.602847 | controller | 16:13:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.602851 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:51 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.602855 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.602859 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.602863 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.602869 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.602873 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.602877 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.602882 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.602886 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.602890 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.602894 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.602898 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.602902 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.602907 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.602911 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.602915 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.602919 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.602923 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.602927 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.602931 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.602936 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.602940 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.602944 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.602955 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.602961 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.602965 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.602969 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.602973 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.602977 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.602983 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.602987 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.602991 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.602995 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.603000 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.603004 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.603008 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.603012 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.603016 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:51 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.603021 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.603025 | controller | 16:13:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.603029 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:53 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.603033 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.603037 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.603041 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.603046 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.603050 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.603054 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.603058 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.603062 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.603066 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.603070 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.603074 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.603078 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.603083 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.603087 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.603091 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.603095 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.603099 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.603103 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.603108 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.603113 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.603117 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.603121 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.603125 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.603130 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.603140 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.603144 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.603148 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.603152 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.603157 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.603161 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.603165 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.603169 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.603173 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.603177 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.603182 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.603186 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.603190 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:53 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.603194 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.603198 | controller | 16:13:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.603202 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:54 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.603207 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.603211 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.603216 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.603221 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.603225 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.603229 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.603233 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.603237 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.603241 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.603246 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.603250 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.603254 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.603259 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.603264 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.603269 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.603273 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.603277 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.603282 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.603286 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.603290 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.603294 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.603298 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.603302 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.603307 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.603311 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.603321 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.603325 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.603329 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.603333 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.603338 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.603342 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.603346 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.603350 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.603354 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.603359 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.603363 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.603367 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:54 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.603371 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.603375 | controller | 16:13:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.603379 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:55 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.603384 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.603388 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.603392 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.603398 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.603402 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.603406 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.603410 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.603415 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.603419 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.603423 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.603427 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.603431 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.603435 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.603440 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.603444 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.603448 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.603452 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.603456 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.603461 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.603466 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.603470 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.603475 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.603481 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.603487 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.603506 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.603512 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.603517 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.603530 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.603536 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.603542 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.603548 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.603553 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.603559 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.603565 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.603569 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.603573 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.603580 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:55 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.603584 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.603588 | controller | 16:13:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.603592 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:56 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.603597 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.603601 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.603605 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.603609 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.603613 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.603618 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.603622 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.603626 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.603630 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.603634 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.603638 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.603642 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.603647 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.603651 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.603655 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.603659 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.603663 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.603667 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.603672 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.603676 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.603680 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.603684 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.603688 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.603704 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.603710 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.603714 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.603719 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.603723 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.603734 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.603741 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.603745 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.603749 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.603754 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.603758 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.603762 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.603766 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.603770 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:56 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.603774 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.603779 | controller | 16:13:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.603783 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:57 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.603787 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.603793 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.603797 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.603801 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.603808 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.603812 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.603816 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.603820 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.603824 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.603828 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.603833 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.603837 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.603841 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.603845 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.603849 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.603853 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.603858 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.603862 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.603866 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.603872 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.603876 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.603880 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.603884 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.603889 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.603893 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.603897 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.603910 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.603914 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.603919 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.603923 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.603933 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.603938 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.603942 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.603946 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.603950 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.603954 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.603958 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:57 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.603963 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.603967 | controller | 16:13:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.603971 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:58 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.603975 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.603979 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.603991 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.603995 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.603999 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.604003 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.604008 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.604012 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.604016 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.604020 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.604024 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.604028 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.604034 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.604038 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.604042 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.604054 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.604059 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.604063 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.604067 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.604071 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.604077 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.604081 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.604085 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.604090 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.604094 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.604098 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.604102 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.604106 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.604110 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.604114 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.604118 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.604128 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.604133 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.604137 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.604141 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.604145 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.604150 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:58 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.604154 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.604158 | controller | 16:13:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.604162 | controller | \"ac-watcher\" not found\n logger.go:42: 16:13:59 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.604166 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.604170 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.604174 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.604182 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.604186 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.604190 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.604194 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.604198 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.604202 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.604207 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.604211 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.604215 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.604219 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.604223 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.604227 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.604231 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.604236 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.604240 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.604244 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.604248 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.604252 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.604257 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.604261 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.604265 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.604269 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.604273 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.604277 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.604281 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.604287 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.604291 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.604296 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.604300 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.604309 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.604314 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.604318 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.604322 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.604328 | controller | expected application_credential_id\"]\n logger.go:42: 16:13:59 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.604332 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.604336 | controller | 16:13:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.604340 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:00 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.604346 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.604350 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.604354 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.604359 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.604363 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.604367 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.604371 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.604375 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.604379 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.604383 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.604387 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.604392 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.604396 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.604400 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.604404 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.604408 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.604412 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.604416 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.604421 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.604426 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.604430 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.604434 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.604439 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.604443 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.604447 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.604451 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.604455 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.604459 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.604465 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.604469 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.604473 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.604477 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.604482 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.604493 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.604499 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.604505 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.604511 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:00 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.604516 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.604522 | controller | 16:14:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.604526 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:02 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.604531 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.604535 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.604539 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.604543 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.604547 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.604551 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.604557 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.604563 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.604569 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.604574 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.604580 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.604586 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.604591 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.604597 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.604603 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.604609 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.604613 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.604617 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.604622 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.604628 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.604632 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.604636 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.604641 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.604645 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.604649 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.604653 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.604657 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.604661 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.604666 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.604670 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.604674 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.604678 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.604682 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.604686 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.604690 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.604716 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.604721 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:02 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.604725 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.604729 | controller | 16:14:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.604733 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:03 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.604738 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.604742 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.604746 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.604750 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.604754 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.604758 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.604762 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.604775 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.604779 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.604783 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.604787 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.604792 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.604798 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.604802 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.604806 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.604810 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.604814 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.604818 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.604823 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.604827 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.604831 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.604835 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.604839 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.604843 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.604847 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.604852 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.604856 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.604860 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.604864 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.604868 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.604872 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.604877 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.604881 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.604885 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.604889 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.604893 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.604903 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:03 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.604908 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.604912 | controller | 16:14:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.604920 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:04 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.604924 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.604928 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.604932 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.604938 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.604942 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.604946 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.604950 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.604954 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.604959 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.604963 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.604967 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.604971 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.604975 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.604979 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.604983 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.604988 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.604992 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.604996 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.605000 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.605004 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.605008 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.605012 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.605017 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.605021 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.605025 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.605029 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.605033 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.605037 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.605041 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.605047 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.605051 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.605056 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.605060 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.605064 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.605068 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.605072 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.605078 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:04 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.605082 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.605092 | controller | 16:14:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.605097 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:05 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.605101 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.605105 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.605110 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.605114 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.605118 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.605122 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.605126 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.605130 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.605135 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.605139 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.605143 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.605147 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.605151 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.605157 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.605161 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.605165 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.605170 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.605174 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.605178 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.605182 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.605186 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.605190 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.605194 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.605199 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.605203 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.605207 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.605211 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.605215 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.605221 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.605225 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.605229 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.605234 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.605238 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.605242 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.605246 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.605250 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.605254 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:05 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.605258 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.605263 | controller | 16:14:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.605272 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:06 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.605277 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.605281 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.605285 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.605289 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.605293 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.605298 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.605302 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.605306 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.605310 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.605314 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.605318 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.605322 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.605327 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.605331 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.605335 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.605339 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.605343 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.605347 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.605351 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.605359 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.605363 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.605367 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.605371 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.605375 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.605379 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.605384 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.605388 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.605392 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.605396 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.605400 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.605404 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.605409 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.605413 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.605417 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.605421 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.605425 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.605429 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:06 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.605435 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.605439 | controller | 16:14:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.605444 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:07 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.605453 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.605458 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.605462 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.605468 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.605472 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.605486 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.605490 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.605494 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.605498 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.605502 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.605506 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.605510 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.605516 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.605521 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.605525 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.605529 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.605533 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.605537 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.605542 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.605546 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.605550 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.605562 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.605566 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.605570 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.605574 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.605579 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.605583 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.605587 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.605591 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.605595 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.605599 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.605604 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.605608 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.605612 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.605616 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.605620 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.605625 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:07 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.605636 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.605640 | controller | 16:14:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.605644 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:08 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.605649 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.605653 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.605662 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.605669 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.605673 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.605677 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.605681 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.605685 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.605689 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.605709 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.605713 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.605718 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.605724 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.605728 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.605732 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.605736 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.605740 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.605745 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.605749 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.605753 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.605757 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.605761 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.605765 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.605770 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.605774 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.605778 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.605782 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.605786 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.605790 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.605795 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.605799 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.605803 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.605807 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.605811 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.605815 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.605820 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.605825 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:08 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.605830 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.605834 | controller | 16:14:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.605838 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:10 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.605842 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.605846 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.605851 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.605861 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.605866 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.605870 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.605875 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.605879 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.605883 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.605887 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.605891 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.605895 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.605899 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.605904 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.605909 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.605913 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.605918 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.605922 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.605926 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.605930 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.605934 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.605938 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.605943 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.605947 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.605951 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.605955 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.605959 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.605963 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.605970 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.605974 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.605979 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.605983 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.605987 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.605991 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.605997 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.606001 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.606005 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:10 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.606010 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.606015 | controller | 16:14:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.606019 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:11 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.606023 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.606027 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.606031 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.606035 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.606045 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.606050 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.606054 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.606058 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.606062 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.606067 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.606071 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.606075 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.606090 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.606095 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.606099 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.606103 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.606107 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.606111 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.606115 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.606121 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.606125 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.606130 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.606134 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.606138 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.606142 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.606146 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.606150 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.606155 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.606159 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.606163 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.606167 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.606171 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.606176 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.606180 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.606184 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.606188 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.606192 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:11 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.606196 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.606201 | controller | 16:14:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.606205 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:12 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.606209 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.606213 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.606217 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.606222 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.606226 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.606230 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.606242 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.606246 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.606250 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.606255 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.606259 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.606264 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.606270 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.606274 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.606278 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.606283 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.606287 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.606291 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.606295 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.606301 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.606305 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.606309 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.606314 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.606318 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.606322 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.606326 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.606330 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.606334 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.606338 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.606344 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.606348 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.606352 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.606357 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.606361 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.606365 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.606369 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.606373 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:12 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.606378 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.606382 | controller | 16:14:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.606386 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:13 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.606390 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.606394 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.606398 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.606404 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.606408 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.606412 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.606417 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.606427 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.606431 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.606435 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.606439 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.606444 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.606448 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.606452 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.606456 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.606460 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.606464 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.606468 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.606472 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.606477 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.606481 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.606485 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.606489 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.606493 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.606498 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.606502 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.606506 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.606510 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.606514 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.606518 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.606522 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.606528 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.606533 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.606537 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.606541 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.606545 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.606551 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:13 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.606555 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.606559 | controller | 16:14:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.606563 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:14 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.606567 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.606572 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.606576 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.606580 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.606584 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.606588 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.606592 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.606596 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.606601 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.606611 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.606615 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.606619 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.606624 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.606628 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.606632 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.606636 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.606640 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.606644 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.606648 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.606653 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.606657 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.606661 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.606665 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.606669 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.606673 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.606678 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.606682 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.606686 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.606701 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.606707 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.606711 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.606716 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.606720 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.606724 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.606728 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.606732 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.606736 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:14 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.606740 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.606745 | controller | 16:14:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.606749 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:15 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.606753 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.606757 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.606761 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.606767 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.606771 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.606776 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.606780 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.606784 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.606788 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.606792 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.606805 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.606809 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.606814 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.606818 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.606822 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.606826 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.606830 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.606834 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.606838 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.606846 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.606850 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.606854 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.606858 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.606862 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.606866 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.606871 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.606875 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.606879 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.606883 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.606887 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.606891 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.606896 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.606900 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.606904 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.606908 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.606912 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.606916 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:15 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.606921 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.606925 | controller | 16:14:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.606929 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:16 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.606933 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.606937 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.606942 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.606946 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.606950 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.606954 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.606958 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.606963 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.606967 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.606971 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.606975 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.606985 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.606990 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.606995 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.606999 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.607003 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.607007 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.607011 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.607016 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.607020 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.607024 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.607028 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.607032 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.607036 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.607040 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.607045 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.607049 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.607055 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.607059 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.607063 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.607067 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.607072 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.607076 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.607080 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.607084 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.607088 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.607092 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:16 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.607097 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.607101 | controller | 16:14:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.607105 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:18 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.607109 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.607113 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.607117 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.607123 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.607127 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.607131 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.607136 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.607140 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.607144 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.607148 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.607152 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.607156 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.607160 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.607170 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.607176 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.607181 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.607185 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.607189 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.607193 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.607199 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.607203 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.607207 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.607211 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.607215 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.607219 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.607223 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.607227 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.607232 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.607236 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.607240 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.607244 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.607248 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.607252 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.607256 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.607261 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.607265 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.607270 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:18 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.607275 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.607279 | controller | 16:14:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.607283 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:19 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.607287 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.607291 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.607296 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.607300 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.607304 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.607308 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.607312 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.607316 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.607321 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.607326 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.607331 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.607335 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.607339 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.607343 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.607353 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.607357 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.607361 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.607366 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.607370 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.607374 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.607378 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.607382 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.607386 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.607391 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.607395 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.607399 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.607403 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.607407 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.607413 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.607417 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.607421 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.607426 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.607430 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.607434 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.607438 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.607442 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.607446 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:19 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.607450 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.607455 | controller | 16:14:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.607459 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:20 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.607463 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.607467 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.607471 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.607475 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.607479 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.607484 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.607488 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.607492 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.607510 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.607515 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.607519 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.607523 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.607527 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.607532 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.607536 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.607540 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.607550 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.607554 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.607558 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.607564 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.607568 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.607573 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.607577 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.607581 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.607585 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.607589 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.607595 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.607599 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.607604 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.607609 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.607613 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.607618 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.607622 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.607626 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.607630 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.607634 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.607639 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:20 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.607643 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.607647 | controller | 16:14:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.607651 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:21 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.607655 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.607659 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.607664 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.607668 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.607672 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.607676 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.607680 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.607684 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.607688 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.607706 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.607712 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.607717 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.607723 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.607727 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.607731 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.607735 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.607739 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.607802 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.607809 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.607813 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.607817 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.607821 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.607826 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.607830 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.607843 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.607847 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.607851 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.607855 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.607859 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.607863 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.607867 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.607872 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.607876 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.607880 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.607884 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.607888 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.607900 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:21 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.607904 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.607908 | controller | 16:14:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.607912 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:22 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.607916 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.607921 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.607925 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.607931 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.607935 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.607939 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.607943 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.607948 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.607954 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.607965 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.607970 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.607974 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.607978 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.607982 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.607986 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.607990 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.607995 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.607999 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.608011 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.608015 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.608019 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.608024 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.608028 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.608032 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.608036 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.608040 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.608044 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.608049 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.608053 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.608057 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.608061 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.608065 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.608069 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.608074 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.608078 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.608082 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.608089 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:22 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.608094 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.608098 | controller | 16:14:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.608102 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:23 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.608106 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.608110 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.608115 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.608120 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.608126 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.608130 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.608134 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.608138 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.608143 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.608147 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.608151 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.608155 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.608159 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.608163 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.608168 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.608172 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.608176 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.608180 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.608184 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.608195 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.608201 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.608205 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.608210 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.608214 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.608218 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.608222 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.608228 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.608232 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.608237 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.608242 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.608246 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.608250 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.608254 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.608259 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.608263 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.608267 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.608271 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:23 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.608275 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.608280 | controller | 16:14:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.608284 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:24 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.608288 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.608292 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.608296 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.608300 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.608304 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.608309 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.608313 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.608317 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.608321 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.608325 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.608329 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.608333 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.608338 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.608342 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.608346 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.608350 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.608354 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.608358 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.608362 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.608369 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.608374 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.608384 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.608389 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.608393 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.608397 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.608402 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.608406 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.608410 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.608414 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.608418 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.608422 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.608426 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.608431 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.608435 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.608439 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.608443 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.608447 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:24 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.608451 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.608456 | controller | 16:14:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.608460 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:26 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.608464 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.608468 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.608472 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.608476 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.608481 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.608485 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.608489 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.608502 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.608507 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.608511 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.608515 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.608519 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.608525 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.608529 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.608534 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.608538 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.608543 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.608547 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.608551 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.608555 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.608559 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.608563 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.608575 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.608579 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.608583 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.608588 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.608592 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.608596 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.608600 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.608604 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.608609 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.608613 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.608617 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.608621 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.608625 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.608629 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.608634 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:26 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.608638 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.608643 | controller | 16:14:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.608655 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:27 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.608659 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.608663 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.608667 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.608673 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.608678 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.608682 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.608686 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.608690 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.608712 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.608716 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.608720 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.608724 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.608729 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.608733 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.608737 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.608741 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.608745 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.608749 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.608754 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.608758 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.608762 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.608766 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.608770 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.608776 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.608786 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.608791 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.608795 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.608799 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.608803 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.608808 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.608812 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.608816 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.608820 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.608824 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.608828 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.608833 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.608838 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:27 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.608843 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.608847 | controller | 16:14:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.608851 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:28 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.608855 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.608859 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.608864 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.608868 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.608872 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.608876 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.608880 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.608884 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.608889 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.608893 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.608897 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.608901 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.608905 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.608909 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.608913 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.608918 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.608922 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.608926 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.608930 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.608934 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.608938 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.608942 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.608947 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.608951 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.608955 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.608965 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.608970 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.608974 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.608980 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.608987 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.608991 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.608995 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.609000 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.609004 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.609008 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.609012 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.609016 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:28 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.609020 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.609025 | controller | 16:14:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.609029 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:29 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.609033 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.609037 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.609043 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.609047 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.609051 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.609055 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.609059 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.609064 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.609068 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.609072 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.609076 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.609080 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.609084 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.609088 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.609093 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.609097 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.609101 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.609105 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.609109 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.609116 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.609120 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.609124 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.609129 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.609133 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.609137 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.609141 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.609145 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.609156 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.609161 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.609165 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.609169 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.609173 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.609178 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.609182 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.609186 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.609190 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.609194 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:29 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.609198 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.609203 | controller | 16:14:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.609207 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:30 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.609211 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.609215 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.609219 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.609224 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.609228 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.609232 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.609236 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.609240 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.609244 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.609248 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.609253 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.609257 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.609262 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.609267 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.609271 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.609275 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.609279 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.609284 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.609288 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.609292 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.609297 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.609302 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.609318 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.609324 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.609328 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.609332 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.609337 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.609341 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.609350 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.609355 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.609359 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.609363 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.609368 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.609372 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.609376 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.609380 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.609384 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:30 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.609389 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.609393 | controller | 16:14:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.609397 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:31 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.609401 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.609405 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.609410 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.609417 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.609421 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.609434 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.609438 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.609442 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.609446 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.609451 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.609455 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.609459 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.609463 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.609467 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.609471 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.609476 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.609480 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.609484 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.609495 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.609500 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.609504 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.609508 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.609512 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.609516 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.609521 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.609525 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.609529 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.609533 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.609537 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.609547 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.609551 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.609562 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.609567 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.609571 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.609575 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.609579 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.609585 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:31 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.609590 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.609594 | controller | 16:14:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.609598 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:33 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.609602 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.609608 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.609612 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.609616 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.609620 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.609625 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.609629 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.609633 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.609637 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.609641 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.609645 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.609649 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.609654 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.609658 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.609662 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.609666 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.609670 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.609674 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.609678 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.609683 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.609687 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.609702 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.609709 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.609713 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.609717 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.609721 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.609725 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.609730 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.609735 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.609740 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.609744 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.609755 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.609759 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.609764 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.609768 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.609772 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.609776 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:33 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.609780 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.609784 | controller | 16:14:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.609789 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:34 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.609793 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.609797 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.609801 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.609805 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.609809 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.609813 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.609818 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.609822 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.609826 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.609830 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.609834 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.609838 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.609842 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.609847 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.609852 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.609857 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.609861 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.609865 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.609869 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.609875 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.609881 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.609885 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.609889 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.609893 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.609897 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.609902 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.609906 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.609910 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.609914 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.609918 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.609922 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.609927 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.609937 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.609941 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.609945 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.609949 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.609954 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:34 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.609958 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.609962 | controller | 16:14:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.609966 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:35 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.609970 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.609975 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.609979 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.609983 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.609987 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.609991 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.609995 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.609999 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.610004 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.610008 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.610012 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.610016 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.610022 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.610026 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.610030 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.610034 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.610038 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.610042 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.610047 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.610054 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.610058 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.610062 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.610066 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.610071 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.610075 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.610079 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.610083 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.610087 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.610092 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.610098 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.610112 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.610118 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.610124 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.610136 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.610141 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.610145 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.610149 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:35 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.610153 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.610158 | controller | 16:14:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.610162 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:36 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.610168 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.610173 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.610177 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.610183 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.610187 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.610191 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.610195 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.610199 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.610204 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.610208 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.610212 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.610216 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.610220 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.610224 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.610228 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.610233 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.610238 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.610244 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.610250 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.610256 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.610261 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.610267 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.610272 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.610278 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.610283 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.610287 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.610291 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.610295 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.610299 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.610306 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.610310 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.610314 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.610319 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.610323 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.610327 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.610340 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.610348 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:36 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.610356 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.610362 | controller | 16:14:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.610368 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:37 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.610373 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.610379 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.610384 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.610388 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.610392 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.610396 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.610400 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.610404 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.610409 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.610413 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.610417 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.610421 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.610425 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.610430 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.610434 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.610440 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.610444 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.610448 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.610452 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.610456 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.610461 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.610465 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.610469 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.610473 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.610477 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.610481 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.610486 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.610490 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.610497 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.610502 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.610508 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.610513 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.610519 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.610523 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.610527 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.610531 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.610542 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:37 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.610547 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.610551 | controller | 16:14:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.610555 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:38 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.610559 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.610563 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.610567 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.610571 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.610576 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.610581 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.610587 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.610592 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.610598 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.610603 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.610607 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.610611 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.610615 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.610619 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.610623 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.610628 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.610633 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.610639 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.610644 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.610652 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.610657 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.610662 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.610668 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.610673 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.610679 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.610683 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.610687 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.610703 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.610711 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.610716 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.610720 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.610724 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.610728 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.610734 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.610739 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.610746 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.610751 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:38 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.610768 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.610774 | controller | 16:14:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.610778 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:40 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.610782 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.610786 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.610790 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.610797 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.610802 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.610808 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.610813 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.610819 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.610825 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.610831 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.610835 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.610839 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.610846 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.610850 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.610854 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.610858 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.610862 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.610867 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.610871 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.610875 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.610879 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.610883 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.610888 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.610892 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.610896 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.610900 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.610904 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.610908 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.610912 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.610917 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.610921 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.610925 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.610929 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.610934 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.610939 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.610945 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.610950 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:40 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.610956 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.610970 | controller | 16:14:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.610975 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:41 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.610980 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.610984 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.610988 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.610995 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.610999 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.611003 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.611008 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.611012 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.611016 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.611020 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.611024 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.611028 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.611033 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.611041 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.611046 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.611052 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.611058 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.611064 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.611069 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.611075 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.611079 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.611083 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.611087 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.611091 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.611096 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.611100 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.611104 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.611108 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.611112 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.611116 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.611121 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.611125 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.611129 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.611133 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.611137 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.611141 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.611150 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:41 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.611154 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.611158 | controller | 16:14:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.611163 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:42 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.611176 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.611181 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.611185 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.611190 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.611194 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.611198 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.611202 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.611206 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.611210 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.611215 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.611219 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.611223 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.611227 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.611231 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.611237 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.611241 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.611246 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.611251 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.611255 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.611259 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.611263 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.611267 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.611271 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.611276 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.611280 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.611284 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.611288 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.611292 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.611298 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.611302 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.611306 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.611310 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.611315 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.611319 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.611323 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.611327 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.611331 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:42 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.611337 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.611341 | controller | 16:14:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.611346 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:43 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.611350 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.611362 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.611366 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.611370 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.611375 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.611379 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.611383 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.611387 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.611391 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.611396 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.611400 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.611404 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.611408 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.611412 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.611416 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.611421 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.611425 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.611429 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.611433 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.611439 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.611443 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.611447 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.611451 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.611456 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.611460 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.611464 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.611468 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.611472 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.611476 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.611481 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.611485 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.611489 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.611493 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.611497 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.611518 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.611524 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.611529 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:43 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.611534 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.611539 | controller | 16:14:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.611543 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:44 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.611547 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.611551 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.611555 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.611566 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.611571 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.611575 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.611579 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.611583 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.611588 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.611592 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.611596 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.611600 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.611608 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.611612 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.611617 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.611621 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.611625 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.611639 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.611643 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.611647 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.611652 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.611656 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.611660 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.611664 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.611668 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.611673 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.611677 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.611681 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.611685 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.611691 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.611718 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.611722 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.611726 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.611731 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.611735 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.611739 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.611743 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:44 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.611747 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.611752 | controller | 16:14:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.611756 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:45 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.611760 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.611764 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.611769 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.611774 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.611786 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.611791 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.611803 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.611807 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.611811 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.611815 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.611819 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.611823 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.611828 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.611832 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.611836 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.611840 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.611844 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.611848 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.611853 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.611857 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.611861 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.611865 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.611869 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.611874 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.611878 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.611882 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.611886 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.611890 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.611894 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.611899 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.611903 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.611907 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.611911 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.611915 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.611921 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.611926 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.611931 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:45 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.611936 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.611940 | controller | 16:14:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.611944 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:47 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.611948 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.611952 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.611957 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.611961 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.611965 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.611969 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.611980 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.611985 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.611989 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.611993 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.611998 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.612002 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.612006 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.612010 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.612014 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.612018 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.612023 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.612027 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.612031 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.612037 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.612041 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.612045 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.612049 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.612054 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.612058 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.612062 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.612066 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.612070 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.612076 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.612080 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.612085 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.612089 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.612093 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.612097 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.612101 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.612105 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.612110 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:47 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.612114 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.612118 | controller | 16:14:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.612122 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:48 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.612126 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.612130 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.612135 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.612140 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.612144 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.612149 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.612153 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.612163 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.612167 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.612172 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.612176 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.612181 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.612186 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.612190 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.612194 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.612198 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.612202 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.612207 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.612211 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.612218 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.612222 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.612226 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.612230 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.612235 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.612239 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.612243 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.612247 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.612251 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.612255 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.612260 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.612264 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.612268 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.612272 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.612276 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.612280 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.612285 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.612289 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:48 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.612293 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.612297 | controller | 16:14:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.612303 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:49 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.612318 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.612323 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.612328 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.612333 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.612337 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.612341 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.612345 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.612349 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.612360 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.612364 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.612369 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.612373 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.612387 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.612391 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.612395 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.612399 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.612403 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.612408 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.612412 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.612416 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.612420 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.612424 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.612428 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.612433 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.612437 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.612441 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.612445 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.612449 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.612453 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.612458 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.612462 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.612468 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.612472 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.612476 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.612480 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.612485 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.612489 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:49 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.612493 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.612497 | controller | 16:14:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.612502 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:50 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.612506 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.612510 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.612514 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.612520 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.612525 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.612530 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.612534 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.612538 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.612542 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.612547 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.612557 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.612562 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.612566 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.612570 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.612574 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.612580 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.612584 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.612589 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.612593 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.612597 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.612601 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.612605 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.612610 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.612614 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.612618 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.612622 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.612626 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.612630 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.612635 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.612639 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.612643 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.612647 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.612652 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.612656 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.612660 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.612664 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.612670 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:50 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.612674 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.612678 | controller | 16:14:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.612682 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:51 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.612686 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.612690 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.612707 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.612712 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.612716 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.612720 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.612724 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.612728 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.612733 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.612737 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.612743 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.612754 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.612758 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.612762 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.612766 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.612771 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.612775 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.612779 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.612783 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.612787 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.612791 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.612796 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.612800 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.612804 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.612808 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.612812 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.612816 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.612821 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.612826 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.612830 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.612835 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.612839 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.612843 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.612847 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.612851 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.612855 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.612859 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:51 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.612864 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.612868 | controller | 16:14:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.612872 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:52 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.612876 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.612880 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.612884 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.612888 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.612893 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.612897 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.612901 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.612905 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.612909 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.612913 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.612918 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.612922 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.612926 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.612936 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.612940 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.612944 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.612949 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.612953 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.612957 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.612964 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.612968 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.612973 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.612977 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.612981 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.612985 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.612989 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.612994 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.612999 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.613004 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.613009 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.613013 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.613017 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.613021 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.613026 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.613030 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.613034 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.613038 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:52 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.613042 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.613046 | controller | 16:14:52 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.613051 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:53 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.613055 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.613059 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.613063 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.613067 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.613072 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.613076 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.613080 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.613084 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.613088 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.613092 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.613096 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.613101 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.613108 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.613112 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.613122 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.613127 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.613131 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.613135 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.613139 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.613144 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.613148 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.613152 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.613156 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.613160 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.613164 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.613168 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.613172 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.613177 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.613181 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.613185 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.613189 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.613193 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.613198 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.613202 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.613206 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.613210 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.613214 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:54 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.613218 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.613223 | controller | 16:14:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.613227 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:55 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.613231 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.613235 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.613239 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.613245 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.613249 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.613253 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.613257 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.613262 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.613266 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.613271 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.613276 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.613280 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.613284 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.613288 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.613292 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.613297 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.613306 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.613311 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.613315 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.613319 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.613324 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.613328 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.613332 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.613336 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.613340 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.613344 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.613348 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.613353 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.613357 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.613361 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.613365 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.613369 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.613373 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.613378 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.613382 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.613386 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.613392 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:55 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.613396 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.613400 | controller | 16:14:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.613404 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:56 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.613408 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.613413 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.613417 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.613423 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.613427 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.613431 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.613435 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.613439 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.613443 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.613448 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.613452 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.613456 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.613460 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.613464 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.613468 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.613472 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.613476 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.613486 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.613491 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.613495 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.613499 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.613503 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.613509 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.613513 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.613518 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.613522 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.613527 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.613531 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.613537 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.613541 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.613545 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.613549 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.613554 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.613558 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.613562 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.613566 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.613570 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:56 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.613574 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.613578 | controller | 16:14:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.613583 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:57 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.613587 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.613591 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.613595 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.613599 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.613603 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.613608 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.613612 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.613616 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.613620 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.613624 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.613628 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.613632 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.613637 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.613641 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.613645 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.613649 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.613653 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.613657 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.613667 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.613673 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.613677 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.613681 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.613685 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.613689 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.613705 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.613710 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.613714 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.613718 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.613723 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.613727 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.613731 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.613735 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.613739 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.613744 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.613748 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.613752 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.613756 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:57 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.613760 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.613764 | controller | 16:14:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.613769 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:58 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.613773 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.613777 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.613781 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.613785 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.613789 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.613794 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.613798 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.613802 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.613808 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.613812 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.613817 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.613821 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.613826 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.613831 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.613836 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.613840 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.613845 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.613849 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.613853 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.613865 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.613870 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.613874 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.613878 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.613883 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.613887 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.613891 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.613895 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.613899 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.613903 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.613907 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.613911 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.613916 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.613920 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.613924 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.613928 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.613932 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.613937 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:58 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.613941 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.613945 | controller | 16:14:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.613949 | controller | \"ac-watcher\" not found\n logger.go:42: 16:14:59 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.613953 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.613957 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.613962 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.613967 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.613971 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.613976 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.613980 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.613984 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.613988 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.613992 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.613996 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.614000 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.614005 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.614009 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.614013 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.614017 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.614021 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.614025 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.614029 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.614034 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.614038 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.614048 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.614053 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.614057 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.614063 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.614067 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.614071 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.614075 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.614080 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.614084 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.614088 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.614092 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.614096 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.614100 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.614104 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.614109 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.614116 | controller | expected application_credential_id\"]\n logger.go:42: 16:14:59 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.614120 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.614124 | controller | 16:14:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.614128 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:00 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.614132 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.614136 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.614141 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.614145 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.614149 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.614153 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.614157 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.614161 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.614165 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.614170 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.614174 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.614178 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.614182 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.614186 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.614190 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.614195 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.614199 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.614203 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.614207 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.614211 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.614215 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.614220 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.614229 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.614234 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.614238 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.614242 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.614246 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.614251 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.614257 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.614262 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.614266 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.614271 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.614275 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.614279 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.614283 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.614287 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.614291 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:00 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.614296 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.614300 | controller | 16:15:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.614304 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:01 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.614308 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.614312 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.614317 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.614321 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.614325 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.614329 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.614333 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.614339 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.614343 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.614347 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.614351 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.614355 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.614359 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.614364 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.614368 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.614372 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.614376 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.614380 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.614384 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.614390 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.614394 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.614398 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.614403 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.614407 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.614417 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.614421 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.614425 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.614429 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.614434 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.614438 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.614442 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.614446 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.614450 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.614454 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.614459 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.614463 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.614467 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:01 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.614471 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.614475 | controller | 16:15:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.614479 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:03 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.614484 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.614488 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.614492 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.614496 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.614500 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.614504 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.614508 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.614513 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.614517 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.614521 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.614525 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.614529 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.614535 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.614539 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.614544 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.614548 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.614552 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.614556 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.614560 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.614564 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.614569 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.614573 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.614577 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.614583 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.614587 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.614597 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.614602 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.614606 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.614610 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.614614 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.614618 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.614623 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.614627 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.614631 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.614635 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.614639 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.614644 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:03 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.614649 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.614654 | controller | 16:15:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.614658 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:04 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.614662 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.614666 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.614670 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.614677 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.614681 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.614686 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.614690 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.614706 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.614710 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.614714 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.614719 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.614723 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.614727 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.614731 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.614735 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.614740 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.614744 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.614748 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.614752 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.614758 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.614762 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.614766 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.614771 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.614775 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.614779 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.614783 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.614794 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.614798 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.614802 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.614806 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.614811 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.614815 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.614819 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.614823 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.614827 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.614832 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.614838 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:04 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.614842 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.614846 | controller | 16:15:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.614850 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:05 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.614854 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.614859 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.614864 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.614868 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.614873 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.614877 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.614881 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.614885 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.614889 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.614893 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.614898 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.614902 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.614907 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.614912 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.614918 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.614923 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.614929 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.614935 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.614939 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.614944 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.614948 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.614952 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.614956 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.614960 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.614964 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.614969 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.614973 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.614977 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.614990 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.614994 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.614998 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.615003 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.615007 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.615012 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.615017 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.615023 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.615029 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:05 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.615034 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.615040 | controller | 16:15:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.615045 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:06 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.615051 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.615055 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.615059 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.615063 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.615068 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.615072 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.615076 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.615080 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.615084 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.615088 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.615093 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.615097 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.615101 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.615105 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.615112 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.615116 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.615120 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.615124 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.615128 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.615136 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.615140 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.615144 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.615150 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.615154 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.615158 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.615162 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.615166 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.615170 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.615175 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.615186 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.615190 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.615195 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.615199 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.615203 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.615207 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.615211 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.615215 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:06 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.615220 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.615224 | controller | 16:15:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.615228 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:07 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.615232 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.615236 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.615241 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.615245 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.615249 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.615254 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.615259 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.615265 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.615270 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.615276 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.615281 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.615287 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.615295 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.615301 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.615306 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.615310 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.615314 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.615318 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.615322 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.615327 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.615331 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.615335 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.615339 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.615343 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.615347 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.615351 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.615356 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.615360 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.615364 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.615368 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.615379 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.615384 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.615388 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.615392 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.615396 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.615400 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.615405 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:07 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.615409 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.615413 | controller | 16:15:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.615417 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:08 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.615421 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.615428 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.615432 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.615438 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.615442 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.615446 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.615450 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.615454 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.615458 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.615463 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.615467 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.615471 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.615476 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.615482 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.615487 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.615493 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.615498 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.615503 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.615527 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.615531 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.615536 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.615540 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.615545 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.615550 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.615555 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.615561 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.615566 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.615572 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.615576 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.615583 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.615587 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.615591 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.615603 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.615608 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.615612 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.615616 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.615622 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:08 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.615627 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.615631 | controller | 16:15:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.615635 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:10 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.615639 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.615643 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.615647 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.615651 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.615656 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.615750 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.615756 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.615760 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.615764 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.615768 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.615773 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.615777 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.615781 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.615785 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.615789 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.615793 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.615798 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.615802 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.615806 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.615813 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.615818 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.615824 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.615830 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.615835 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.615841 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.615847 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.615853 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.615858 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.615865 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.615869 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.615873 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.615878 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.615882 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.615896 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.615900 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.615904 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.615909 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:10 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.615914 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.615920 | controller | 16:15:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.615926 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:11 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.615931 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.615937 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.615942 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.615948 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.615953 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.615959 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.615965 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.615970 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.615975 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.615979 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.615983 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.615987 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.615992 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.615996 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.616003 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.616007 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.616011 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.616015 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.616019 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.616026 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.616030 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.616034 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.616038 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.616042 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.616047 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.616051 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.616055 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.616059 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.616063 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.616068 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.616072 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.616076 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.616080 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.616085 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.616097 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.616102 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.616106 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:11 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.616110 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.616115 | controller | 16:15:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.616119 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:12 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.616125 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.616129 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.616133 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.616141 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.616146 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.616152 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.616157 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.616163 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.616168 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.616174 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.616179 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.616183 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.616189 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.616193 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.616197 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.616201 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.616207 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.616213 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.616218 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.616224 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.616229 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.616233 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.616237 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.616241 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.616245 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.616250 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.616254 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.616258 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.616262 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.616266 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.616271 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.616275 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.616279 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.616283 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.616287 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.616292 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.616304 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:12 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.616309 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.616313 | controller | 16:15:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.616317 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:13 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.616322 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.616326 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.616330 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.616338 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.616342 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.616346 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.616350 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.616355 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.616359 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.616363 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.616367 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.616371 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.616375 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.616380 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.616384 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.616391 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.616397 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.616403 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.616409 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.616413 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.616418 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.616422 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.616426 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.616430 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.616434 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.616438 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.616443 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.616447 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.616451 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.616455 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.616459 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.616464 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.616468 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.616472 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.616476 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.616480 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.616486 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:13 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.616498 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.616502 | controller | 16:15:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.616507 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:14 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.616511 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.616515 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.616519 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.616523 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.616528 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.616532 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.616536 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.616540 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.616544 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.616548 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.616553 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.616557 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.616561 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.616565 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.616571 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.616575 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.616579 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.616584 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.616588 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.616592 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.616596 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.616600 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.616605 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.616609 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.616613 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.616617 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.616621 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.616625 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.616631 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.616635 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.616640 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.616644 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.616648 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.616652 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.616656 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.616661 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.616665 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:14 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.616669 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.616681 | controller | 16:15:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.616686 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:15 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.616690 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.616706 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.616710 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.616714 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.616718 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.616723 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.616727 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.616731 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.616735 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.616739 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.616743 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.616747 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.616752 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.616756 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.616760 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.616764 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.616768 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.616772 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.616777 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.616784 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.616788 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.616793 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.616797 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.616801 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.616805 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.616809 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.616813 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.616818 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.616822 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.616826 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.616830 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.616834 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.616838 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.616843 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.616847 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.616851 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.616855 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:15 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.616859 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.616864 | controller | 16:15:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.616868 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:16 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.616879 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.616883 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.616887 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.616892 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.616898 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.616902 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.616906 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.616910 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.616914 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.616919 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.616923 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.616927 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.616933 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.616937 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.616941 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.616946 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.616951 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.616955 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.616959 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.616963 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.616968 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.616972 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.616976 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.616980 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.616984 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.616988 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.616993 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.616997 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.617001 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.617005 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.617009 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.617015 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.617019 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.617024 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.617028 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.617032 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.617036 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:16 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.617040 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.617045 | controller | 16:15:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.617049 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:18 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.617053 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.617065 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.617069 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.617075 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.617095 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.617099 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.617104 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.617108 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.617112 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.617116 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.617120 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.617124 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.617129 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.617133 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.617137 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.617141 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.617145 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.617150 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.617154 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.617158 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.617162 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.617166 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.617170 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.617175 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.617179 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.617183 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.617187 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.617191 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.617195 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.617200 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.617204 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.617208 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.617212 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.617216 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.617221 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.617225 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.617230 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:18 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.617236 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.617240 | controller | 16:15:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.617245 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:19 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.617249 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.617274 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.617279 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.617290 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.617295 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.617299 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.617303 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.617307 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.617311 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.617316 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.617320 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.617324 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.617328 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.617332 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.617336 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.617340 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.617344 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.617349 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.617353 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.617357 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.617361 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.617365 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.617369 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.617374 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.617378 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.617382 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.617386 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.617390 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.617399 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.617404 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.617410 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.617415 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.617421 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.617426 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.617430 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.617434 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.617439 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:19 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.617443 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.617447 | controller | 16:15:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.617451 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:20 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.617455 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.617459 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.617464 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.617470 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.617482 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.617486 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.617490 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.617495 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.617499 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.617503 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.617507 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.617511 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.617517 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.617521 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.617526 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.617530 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.617534 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.617538 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.617542 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.617548 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.617552 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.617556 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.617561 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.617565 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.617569 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.617573 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.617577 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.617581 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.617585 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.617590 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.617594 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.617598 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.617602 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.617606 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.617611 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.617615 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.617619 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:20 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.617623 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.617627 | controller | 16:15:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.617632 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:21 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.617636 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.617640 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.617644 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.617648 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.617652 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.617663 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.617668 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.617672 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.617676 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.617680 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.617685 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.617689 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.617717 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.617724 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.617728 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.617732 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.617737 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.617741 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.617745 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.617751 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.617755 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.617760 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.617764 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.617768 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.617772 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.617776 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.617780 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.617785 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.617789 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.617793 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.617797 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.617801 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.617805 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.617810 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.617815 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.617819 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.617824 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:21 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.617828 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.617832 | controller | 16:15:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.617836 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:22 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.617840 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.617845 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.617849 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.617855 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.617859 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.617863 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.617867 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.617879 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.617883 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.617888 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.617892 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.617896 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.617900 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.617905 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.617911 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.617915 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.617919 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.617924 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.617928 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.617932 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.617936 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.617941 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.617945 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.617949 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.617953 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.617957 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.617962 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.617966 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.617971 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.617977 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.617982 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.617989 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.617994 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.618000 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.618006 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.618010 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.618017 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:22 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.618023 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.618028 | controller | 16:15:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.618034 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:23 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.618040 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.618046 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.618051 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.618055 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.618060 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.618064 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.618068 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.618072 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.618085 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.618090 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.618094 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.618104 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.618109 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.618113 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.618117 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.618122 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.618128 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.618133 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.618139 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.618145 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.618150 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.618154 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.618158 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.618163 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.618167 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.618171 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.618175 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.618181 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.618194 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.618200 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.618204 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.618208 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.618213 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.618217 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.618221 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.618225 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.618230 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:23 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.618235 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.618241 | controller | 16:15:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.618247 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:25 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.618253 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.618257 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.618261 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.618266 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.618270 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.618274 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.618279 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.618285 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.618290 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.618296 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.618312 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.618317 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.618321 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.618326 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.618330 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.618334 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.618338 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.618343 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.618347 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.618353 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.618357 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.618362 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.618366 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.618370 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.618374 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.618380 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.618385 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.618391 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.618396 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.618405 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.618410 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.618416 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.618420 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.618425 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.618429 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.618433 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.618437 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:25 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.618442 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.618446 | controller | 16:15:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.618450 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:26 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.618454 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.618459 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.618463 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.618467 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.618471 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.618476 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.618483 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.618487 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.618491 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.618497 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.618503 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.618519 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.618528 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.618532 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.618537 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.618541 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.618545 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.618550 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.618554 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.618558 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.618562 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.618567 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.618571 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.618575 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.618579 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.618584 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.618588 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.618592 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.618596 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.618601 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.618605 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.618609 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.618613 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.618617 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.618622 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.618626 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.618631 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:26 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.618636 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.618642 | controller | 16:15:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.618648 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:27 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.618654 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.618659 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.618665 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.618672 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.618676 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.618680 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.618685 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.618689 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.618721 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.618727 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.618734 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.618738 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.618742 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.618758 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.618764 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.618770 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.618775 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.618780 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.618786 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.618794 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.618800 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.618805 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.618811 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.618817 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.618823 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.618829 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.618833 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.618838 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.618842 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.618846 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.618851 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.618855 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.618859 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.618863 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.618868 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.618872 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.618879 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:27 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.618883 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.618887 | controller | 16:15:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.618892 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:28 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.618897 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.618902 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.618908 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.618916 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.618921 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.618925 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.618929 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.618934 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.618938 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.618942 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.618946 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.618951 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.618955 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.618959 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.618972 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.618977 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.618983 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.618989 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.618995 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.619000 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.619004 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.619009 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.619013 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.619017 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.619022 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.619026 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.619030 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.619036 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.619042 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.619047 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.619051 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.619055 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.619059 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.619064 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.619068 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.619072 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.619076 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:28 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.619081 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.619085 | controller | 16:15:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.619089 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:29 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.619093 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.619098 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.619102 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.619106 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.619110 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.619115 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.619120 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.619126 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.619132 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.619138 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.619143 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.619148 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.619153 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.619157 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.619161 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.619165 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.619179 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.619183 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.619188 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.619194 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.619198 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.619202 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.619207 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.619211 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.619215 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.619219 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.619223 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.619227 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.619232 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.619236 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.619240 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.619244 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.619248 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.619253 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.619257 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.619261 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.619265 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:29 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.619271 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.619276 | controller | 16:15:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.619280 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:30 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.619284 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.619288 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.619293 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.619297 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.619301 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.619305 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.619309 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.619314 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.619318 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.619324 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.619328 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.619332 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.619338 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.619342 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.619347 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.619352 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.619356 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.619367 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.619372 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.619376 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.619381 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.619385 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.619389 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.619393 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.619397 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.619401 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.619406 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.619410 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.619414 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.619418 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.619422 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.619426 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.619431 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.619435 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.619439 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.619443 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.619447 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:30 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.619451 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.619456 | controller | 16:15:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.619460 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:31 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.619464 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.619468 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.619472 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.619478 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.619482 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.619487 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.619491 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.619495 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.619499 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.619503 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.619508 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.619530 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.619535 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.619540 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.619545 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.619550 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.619555 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.619559 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.619573 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.619578 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.619583 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.619588 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.619593 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.619598 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.619602 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.619607 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.619616 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.619621 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.619626 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.619631 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.619636 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.619641 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.619646 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.619651 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.619656 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.619660 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.619668 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:31 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.619673 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.619678 | controller | 16:15:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.619682 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:33 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.619687 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.619692 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.619713 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.619718 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.619723 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.619728 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.619733 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.619737 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.619742 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.619747 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.619752 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.619757 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.619762 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.619767 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.619772 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.619776 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.619782 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.619787 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.619793 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.619810 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.619816 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.619820 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.619824 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.619829 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.619833 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.619837 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.619841 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.619845 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.619851 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.619857 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.619861 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.619866 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.619870 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.619874 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.619878 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.619882 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.619888 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:33 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.619894 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.619899 | controller | 16:15:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.619905 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:34 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.619911 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.619916 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.619921 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.619925 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.619929 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.619934 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.619939 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.619945 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.619952 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.619958 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.619963 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.619967 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.619972 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.619978 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.619983 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.619989 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.619994 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.620000 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.620004 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.620010 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.620023 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.620028 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.620032 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.620037 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.620041 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.620045 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.620050 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.620056 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.620061 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.620067 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.620072 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.620078 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.620083 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.620087 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.620091 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.620095 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.620099 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:34 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.620103 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.620108 | controller | 16:15:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.620112 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:35 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.620116 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.620120 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.620124 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.620129 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.620133 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.620137 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.620141 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.620145 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.620149 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.620154 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.620160 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.620165 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.620173 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.620178 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.620183 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.620187 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.620191 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.620197 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.620202 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.620208 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.620213 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.620217 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.620230 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.620235 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.620242 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.620247 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.620253 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.620258 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.620264 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.620268 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.620273 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.620277 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.620281 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.620285 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.620289 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.620293 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.620298 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:35 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.620302 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.620306 | controller | 16:15:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.620310 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:36 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.620314 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.620318 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.620323 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.620333 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.620339 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.620344 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.620350 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.620356 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.620361 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.620367 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.620372 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.620377 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.620381 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.620385 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.620389 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.620393 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.620398 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.620402 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.620406 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.620412 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.620418 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.620423 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.620428 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.620443 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.620450 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.620456 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.620462 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.620468 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.620474 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.620479 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.620485 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.620489 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.620493 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.620497 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.620501 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.620506 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.620514 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:36 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.620518 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.620522 | controller | 16:15:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.620526 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:37 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.620531 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.620535 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.620539 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.620543 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.620547 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.620551 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.620556 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.620562 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.620566 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.620570 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.620575 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.620579 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.620583 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.620587 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.620591 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.620595 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.620599 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.620604 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.620608 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.620612 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.620616 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.620621 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.620627 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.620632 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.620637 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.620652 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.620656 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.620661 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.620667 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.620671 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.620675 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.620680 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.620684 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.620688 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.620704 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.620712 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.620716 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:37 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.620721 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.620725 | controller | 16:15:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.620729 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:38 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.620733 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.620737 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.620741 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.620745 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.620750 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.620754 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.620758 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.620762 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.620766 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.620770 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.620775 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.620779 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.620783 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.620787 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.620793 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.620798 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.620802 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.620806 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.620810 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.620819 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.620823 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.620827 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.620831 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.620837 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.620841 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.620845 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.620858 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.620863 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.620867 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.620871 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.620875 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.620879 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.620884 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.620888 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.620892 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.620896 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.620900 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:38 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.620904 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.620908 | controller | 16:15:38 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.620913 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:39 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.620917 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.620921 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.620925 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.620929 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.620933 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.620937 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.620942 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.620946 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.620950 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.620954 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.620958 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.620962 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.620968 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.620972 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.620977 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.620981 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.620985 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.620989 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.620993 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.620997 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.621002 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.621006 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.621010 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.621014 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.621018 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.621022 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.621026 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.621030 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.621043 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.621047 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.621051 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.621056 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.621060 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.621064 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.621068 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.621073 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.621077 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:39 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.621081 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.621085 | controller | 16:15:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.621089 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:41 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.621094 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.621098 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.621104 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.621110 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.621114 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.621118 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.621123 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.621127 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.621131 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.621135 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.621139 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.621143 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.621147 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.621152 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.621156 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.621160 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.621164 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.621168 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.621172 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.621177 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.621181 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.621185 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.621189 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.621193 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.621198 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.621202 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.621206 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.621210 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.621214 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.621227 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.621232 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.621236 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.621241 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.621245 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.621249 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.621253 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.621260 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:41 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.621264 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.621268 | controller | 16:15:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.621273 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:42 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.621277 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.621281 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.621285 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.621289 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.621293 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.621298 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.621302 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.621306 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.621310 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.621314 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.621319 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.621323 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.621327 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.621331 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.621336 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.621340 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.621344 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.621348 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.621352 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.621356 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.621361 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.621365 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.621371 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.621375 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.621379 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.621383 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.621387 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.621392 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.621397 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.621401 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.621412 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.621416 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.621420 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.621424 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.621429 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.621433 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.621438 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:42 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.621442 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.621447 | controller | 16:15:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.621451 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:43 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.621455 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.621459 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.621465 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.621470 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.621479 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.621485 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.621490 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.621496 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.621501 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.621505 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.621510 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.621514 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.621518 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.621522 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.621526 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.621531 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.621535 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.621539 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.621543 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.621549 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.621555 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.621559 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.621564 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.621568 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.621572 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.621576 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.621580 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.621584 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.621589 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.621593 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.621597 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.621601 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.621613 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.621618 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.621622 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.621626 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.621630 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:43 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.621635 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.621639 | controller | 16:15:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.621643 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:44 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.621647 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.621653 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.621657 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.621663 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.621667 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.621671 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.621675 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.621679 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.621683 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.621688 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.621703 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.621710 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.621717 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.621721 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.621726 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.621730 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.621734 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.621738 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.621742 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.621748 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.621753 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.621757 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.621761 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.621765 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.621769 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.621774 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.621778 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.621782 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.621786 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.621790 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.621794 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.621799 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.621803 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.621814 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.621820 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.621824 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.621828 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:44 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.621832 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.621836 | controller | 16:15:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.621841 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:45 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.621845 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.621849 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.621853 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.621859 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.621863 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.621867 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.621872 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.621876 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.621880 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.621884 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.621888 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.621893 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.621897 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.621901 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.621905 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.621909 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.621913 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.621918 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.621922 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.621926 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.621932 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.621937 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.621941 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.621946 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.621950 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.621954 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.621958 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.621962 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.621966 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.621971 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.621975 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.621979 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.621983 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.621987 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.621997 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.622002 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.622008 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:45 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.622012 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.622016 | controller | 16:15:45 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.622020 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:46 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.622024 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.622029 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.622033 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.622037 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.622041 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.622045 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.622050 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.622054 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.622058 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.622062 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.622066 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.622070 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.622075 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.622079 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.622085 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.622090 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.622094 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.622098 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.622102 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.622107 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.622111 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.622115 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.622119 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.622123 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.622127 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.622131 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.622136 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.622140 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.622145 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.622150 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.622154 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.622158 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.622162 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.622166 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.622170 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.622175 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.622185 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:46 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.622190 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.622194 | controller | 16:15:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.622198 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:48 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.622208 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.622213 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.622217 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.622221 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.622225 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.622229 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.622233 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.622238 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.622242 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.622246 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.622250 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.622254 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.622258 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.622263 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.622267 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.622271 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.622275 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.622279 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.622283 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.622289 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.622293 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.622297 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.622302 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.622306 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.622310 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.622314 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.622318 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.622322 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.622326 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.622331 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.622335 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.622339 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.622343 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.622347 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.622352 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.622356 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.622360 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:48 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.622370 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.622374 | controller | 16:15:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.622379 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:49 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.622383 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.622387 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.622391 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.622395 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.622399 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.622404 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.622408 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.622412 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.622416 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.622420 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.622424 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.622428 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.622434 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.622438 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.622443 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.622449 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.622454 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.622458 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.622462 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.622466 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.622470 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.622474 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.622479 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.622483 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.622487 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.622491 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.622495 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.622499 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.622504 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.622509 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.622513 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.622518 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.622522 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.622526 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.622530 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.622535 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.622539 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:49 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.622543 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.622554 | controller | 16:15:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.622559 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:50 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.622563 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.622567 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.622571 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.622577 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.622581 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.622586 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.622590 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.622594 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.622598 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.622602 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.622606 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.622610 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.622615 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.622619 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.622623 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.622627 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.622631 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.622635 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.622640 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.622645 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.622650 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.622654 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.622658 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.622662 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.622666 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.622670 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.622674 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.622679 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.622683 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.622687 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.622702 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.622709 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.622713 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.622717 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.622721 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.622726 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.622731 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:50 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.622736 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.622740 | controller | 16:15:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.622748 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:51 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.622759 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.622763 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.622768 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.622772 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.622776 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.622780 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.622784 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.622788 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.622793 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.622797 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.622801 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.622805 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.622809 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.622814 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.622818 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.622822 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.622826 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.622830 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.622834 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.622839 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.622843 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.622847 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.622851 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.622855 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.622859 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.622864 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.622868 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.622872 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.622879 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.622883 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.622888 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.622892 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.622896 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.622900 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.622904 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.622908 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.622913 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:51 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.622917 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.622921 | controller | 16:15:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.622925 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:52 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.622929 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.622939 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.622943 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.622950 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.622954 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.622959 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.622963 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.622967 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.622971 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.622975 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.622979 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.622984 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.622988 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.622993 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.622998 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.623002 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.623006 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.623010 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.623014 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.623020 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.623024 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.623028 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.623032 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.623037 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.623041 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.623045 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.623049 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.623053 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.623058 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.623062 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.623066 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.623070 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.623074 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.623079 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.623083 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.623087 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.623091 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:52 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.623095 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.623099 | controller | 16:15:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.623104 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:54 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.623108 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.623112 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.623116 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.623127 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.623132 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.623136 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.623140 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.623145 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.623149 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.623153 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.623157 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.623161 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.623167 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.623171 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.623175 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.623180 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.623184 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.623188 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.623192 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.623196 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.623201 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.623205 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.623209 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.623213 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.623217 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.623221 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.623225 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.623230 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.623234 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.623238 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.623242 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.623246 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.623251 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.623255 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.623259 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.623263 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.623267 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:54 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.623274 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.623278 | controller | 16:15:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.623282 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:55 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.623287 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.623291 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.623295 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.623301 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.623311 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.623316 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.623320 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.623324 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.623328 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.623332 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.623337 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.623341 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.623345 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.623349 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.623355 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.623359 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.623363 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.623367 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.623371 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.623376 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.623380 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.623384 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.623388 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.623392 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.623396 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.623400 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.623405 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.623409 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.623413 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.623417 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.623423 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.623427 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.623431 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.623436 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.623440 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.623444 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.623450 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:55 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.623454 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.623458 | controller | 16:15:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.623462 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:56 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.623466 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.623470 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.623475 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.623479 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.623483 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.623494 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.623498 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.623503 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.623507 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.623512 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.623536 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.623541 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.623548 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.623553 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.623558 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.623562 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.623567 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.623572 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.623577 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.623583 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.623588 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.623593 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.623598 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.623603 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.623608 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.623613 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.623617 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.623622 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.623629 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.623634 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.623639 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.623644 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.623649 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.623653 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.623658 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.623663 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.623668 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:56 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.623675 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.623679 | controller | 16:15:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.623684 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:57 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.623689 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.623707 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.623714 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.623719 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.623724 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.623729 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.623734 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.623747 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.623752 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.623757 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.623762 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.623767 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.623773 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.623777 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.623781 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.623786 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.623790 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.623794 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.623798 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.623805 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.623810 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.623814 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.623818 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.623822 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.623826 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.623830 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.623834 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.623839 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.623843 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.623849 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.623853 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.623857 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.623861 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.623865 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.623871 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.623875 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.623879 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:57 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.623884 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.623888 | controller | 16:15:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.623892 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:58 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.623896 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.623900 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.623905 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.623909 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.623913 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.623917 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.623921 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.623926 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.623937 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.623941 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.623946 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.623950 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.623956 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.623960 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.623964 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.623968 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.623972 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.623977 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.623981 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.623985 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.623989 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.623993 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.623997 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.624002 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.624006 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.624010 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.624014 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.624018 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.624022 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.624027 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.624031 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.624035 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.624039 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.624043 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.624048 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.624052 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.624056 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:58 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.624060 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.624064 | controller | 16:15:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.624069 | controller | \"ac-watcher\" not found\n logger.go:42: 16:15:59 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.624073 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.624077 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.624081 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.624087 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.624091 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.624095 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.624099 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.624104 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.624108 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.624112 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.624124 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.624129 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.624133 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.624137 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.624142 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.624146 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.624150 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.624154 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.624158 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.624162 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.624166 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.624171 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.624175 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.624179 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.624183 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.624187 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.624191 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.624195 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.624200 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.624204 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.624208 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.624212 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.624216 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.624221 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.624225 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.624229 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.624235 | controller | expected application_credential_id\"]\n logger.go:42: 16:15:59 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.624239 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.624243 | controller | 16:15:59 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.624247 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:00 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.624252 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.624256 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.624260 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.624266 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.624270 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.624274 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.624278 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.624282 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.624287 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.624291 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.624295 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.624305 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.624310 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.624314 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.624318 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.624323 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.624327 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.624331 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.624335 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.624339 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.624343 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.624347 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.624351 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.624356 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.624360 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.624364 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.624368 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.624372 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.624378 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.624382 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.624386 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.624392 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.624396 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.624400 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.624405 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.624409 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.624413 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:00 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.624417 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.624422 | controller | 16:16:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.624426 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:02 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.624430 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.624434 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.624438 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.624442 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.624447 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.624451 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.624455 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.624459 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.624463 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.624467 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.624472 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.624476 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.624486 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.624490 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.624495 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.624499 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.624503 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.624507 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.624512 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.624519 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.624523 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.624527 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.624531 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.624535 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.624539 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.624544 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.624548 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.624552 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.624556 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.624560 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.624564 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.624569 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.624573 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.624577 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.624581 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.624585 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.624590 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:02 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.624594 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.624598 | controller | 16:16:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.624602 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:03 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.624606 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.624611 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.624615 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.624619 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.624623 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.624627 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.624632 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.624636 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.624640 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.624644 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.624650 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.624654 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.624659 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.624664 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.624676 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.624680 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.624685 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.624689 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.624706 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.624710 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.624714 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.624718 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.624723 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.624727 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.624731 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.624735 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.624739 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.624743 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.624748 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.624752 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.624756 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.624760 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.624764 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.624769 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.624773 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.624777 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.624781 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:03 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.624785 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.624789 | controller | 16:16:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.624794 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:04 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.624798 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.624802 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.624806 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.624813 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.624818 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.624822 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.624826 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.624831 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.624836 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.624841 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.624847 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.624853 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.624858 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.624864 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.624869 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.624882 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.624886 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.624890 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.624894 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.624899 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.624903 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.624907 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.624911 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.624915 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.624920 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.624924 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.624928 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.624936 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.624942 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.624947 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.624953 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.624959 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.624964 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.624970 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.624975 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.624979 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.624985 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:04 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.624989 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.624993 | controller | 16:16:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.624998 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:05 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.625002 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.625006 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.625010 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.625014 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.625018 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.625023 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.625029 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.625035 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.625041 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.625046 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.625052 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.625056 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.625060 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.625065 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.625069 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.625075 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.625080 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.625094 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.625101 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.625105 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.625110 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.625116 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.625121 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.625127 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.625132 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.625138 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.625142 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.625146 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.625153 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.625159 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.625163 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.625167 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.625171 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.625176 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.625180 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.625184 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.625189 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:05 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.625195 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.625200 | controller | 16:16:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.625206 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:06 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.625211 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.625217 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.625223 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.625228 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.625234 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.625239 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.625245 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.625249 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.625253 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.625260 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.625264 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.625268 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.625272 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.625276 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.625282 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.625287 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.625293 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.625298 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.625313 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.625320 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.625325 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.625329 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.625333 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.625337 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.625342 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.625346 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.625350 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.625354 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.625360 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.625365 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.625370 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.625376 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.625381 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.625387 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.625393 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.625398 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.625404 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:06 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.625409 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.625414 | controller | 16:16:06 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.625418 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:07 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.625422 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.625426 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.625430 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.625435 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.625439 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.625443 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.625447 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.625451 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.625455 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.625460 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.625464 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.625468 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.625474 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.625479 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.625485 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.625491 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.625496 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.625502 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.625508 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.625525 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.625531 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.625535 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.625539 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.625543 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.625548 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.625552 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.625557 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.625561 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.625566 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.625570 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.625574 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.625578 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.625582 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.625587 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.625591 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.625595 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.625599 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:07 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.625603 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.625607 | controller | 16:16:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.625612 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:08 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.625616 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.625620 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.625624 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.625632 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.625636 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.625640 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.625645 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.625650 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.625655 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.625661 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.625667 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.625671 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.625675 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.625679 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.625684 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.625688 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.625705 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.625712 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.625716 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.625721 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.625733 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.625738 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.625742 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.625747 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.625751 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.625755 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.625759 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.625763 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.625767 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.625772 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.625776 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.625780 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.625784 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.625788 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.625793 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.625797 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.625803 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:08 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.625807 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.625812 | controller | 16:16:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.625816 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:10 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.625820 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.625824 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.625828 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.625833 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.625839 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.625843 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.625848 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.625852 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.625856 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.625860 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.625864 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.625869 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.625873 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.625877 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.625881 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.625887 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.625891 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.625895 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.625899 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.625904 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.625908 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.625912 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.625926 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.625931 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.625935 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.625939 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.625943 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.625948 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.625953 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.625957 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.625961 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.625966 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.625970 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.625974 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.625978 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.625982 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.625986 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:10 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.625991 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.625995 | controller | 16:16:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.625999 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:11 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.626003 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.626007 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.626012 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.626016 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.626020 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.626024 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.626028 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.626032 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.626037 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.626041 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.626045 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.626049 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.626053 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.626058 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.626063 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.626067 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.626072 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.626076 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.626080 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.626085 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.626090 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.626094 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.626098 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.626109 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.626115 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.626119 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.626123 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.626128 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.626132 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.626136 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.626140 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.626144 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.626148 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.626152 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.626157 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.626161 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.626165 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:11 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.626169 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.626173 | controller | 16:16:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.626177 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:12 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.626182 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.626186 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.626190 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.626194 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.626198 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.626203 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.626207 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.626211 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.626215 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.626219 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.626223 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.626227 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.626233 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.626237 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.626241 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.626245 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.626250 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.626254 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.626258 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.626262 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.626266 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.626270 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.626275 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.626279 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.626283 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.626293 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.626297 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.626302 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.626306 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.626310 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.626314 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.626318 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.626323 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.626327 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.626331 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.626335 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.626339 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:12 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.626343 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.626348 | controller | 16:16:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.626352 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:13 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.626356 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.626360 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.626365 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.626371 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.626375 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.626379 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.626383 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.626389 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.626393 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.626397 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.626401 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.626405 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.626410 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.626414 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.626418 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.626422 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.626426 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.626430 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.626435 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.626440 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.626445 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.626449 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.626453 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.626457 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.626461 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.626465 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.626476 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.626481 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.626485 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.626491 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.626495 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.626499 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.626503 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.626508 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.626512 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.626516 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.626525 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:13 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.626529 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.626534 | controller | 16:16:13 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.626538 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:14 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.626542 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.626546 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.626550 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.626554 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.626558 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.626563 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.626567 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.626571 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.626575 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.626579 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.626583 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.626588 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.626592 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.626596 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.626600 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.626604 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.626608 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.626612 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.626617 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.626621 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.626625 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.626629 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.626633 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.626639 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.626643 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.626647 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.626652 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.626656 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.626667 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.626672 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.626676 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.626680 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.626684 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.626689 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.626703 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.626709 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.626714 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:14 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.626718 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.626722 | controller | 16:16:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.626726 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:15 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.626730 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.626734 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.626739 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.626743 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.626747 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.626751 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.626755 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.626759 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.626764 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.626768 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.626772 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.626776 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.626780 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.626784 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.626788 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.626792 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.626797 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.626801 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.626805 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.626811 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.626815 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.626819 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.626823 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.626827 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.626831 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.626836 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.626840 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.626844 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.626848 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.626858 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.626862 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.626866 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.626871 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.626875 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.626879 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.626883 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.626887 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:15 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.626891 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.626896 | controller | 16:16:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.626900 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:17 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.626904 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.626908 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.626914 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.626925 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.626929 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.626933 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.626938 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.626942 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.626946 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.626950 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.626954 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.626958 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.626964 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.626968 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.626973 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.626977 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.626981 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.626985 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.626989 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.626993 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.626998 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.627002 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.627006 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.627010 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.627014 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.627018 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.627023 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.627027 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.627031 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.627035 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.627046 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.627051 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.627063 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.627068 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.627072 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.627076 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.627080 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:17 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.627084 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.627089 | controller | 16:16:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.627093 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:18 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.627097 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.627101 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.627105 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.627111 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.627115 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.627119 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.627124 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.627128 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.627132 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.627136 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.627140 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.627144 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.627149 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.627153 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.627157 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.627161 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.627165 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.627169 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.627173 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.627178 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.627182 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.627186 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.627192 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.627196 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.627200 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.627204 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.627208 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.627213 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.627217 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.627221 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.627225 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.627229 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.627240 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.627245 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.627249 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.627253 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.627259 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:18 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.627263 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.627267 | controller | 16:16:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.627271 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:19 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.627276 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.627280 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.627284 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.627288 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.627292 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.627297 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.627301 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.627305 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.627309 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.627313 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.627317 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.627322 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.627326 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.627330 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.627336 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.627340 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.627344 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.627348 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.627352 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.627358 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.627362 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.627366 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.627370 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.627375 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.627379 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.627383 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.627387 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.627391 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.627397 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.627401 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.627405 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.627409 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.627414 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.627424 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.627428 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.627432 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.627436 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:19 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.627441 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.627445 | controller | 16:16:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.627449 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:20 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.627453 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.627459 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.627463 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.627467 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.627472 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.627476 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.627480 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.627484 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.627488 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.627492 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.627496 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.627500 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.627505 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.627509 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.627513 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.627602 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.627608 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.627613 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.627618 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.627625 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.627630 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.627635 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.627640 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.627645 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.627650 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.627655 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.627659 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.627664 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.627669 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.627674 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.627679 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.627684 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.627689 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.627705 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.627722 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.627727 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.627732 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:20 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.627737 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.627742 | controller | 16:16:20 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.627747 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:21 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.627752 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.627757 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.627761 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.627766 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.627771 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.627777 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.627781 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.627785 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.627790 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.627794 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.627798 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.627802 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.627808 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.627812 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.627816 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.627821 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.627825 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.627829 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.627833 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.627837 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.627844 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.627848 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.627852 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.627856 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.627860 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.627865 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.627869 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.627873 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.627877 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.627883 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.627887 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.627891 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.627895 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.627899 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.627904 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.627908 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.627919 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:21 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.627923 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.627927 | controller | 16:16:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.627932 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:22 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.627936 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.627940 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.627944 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.627950 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.627954 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.627958 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.627963 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.627967 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.627971 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.627975 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.627979 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.627984 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.627988 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.627992 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.627996 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.628000 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.628004 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.628008 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.628013 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.628017 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.628021 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.628025 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.628029 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.628034 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.628038 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.628042 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.628046 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.628050 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.628054 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.628058 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.628063 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.628067 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.628071 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.628075 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.628079 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.628083 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.628089 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:22 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.628104 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.628109 | controller | 16:16:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.628113 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:23 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.628119 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.628123 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.628127 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.628131 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.628135 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.628140 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.628144 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.628148 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.628152 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.628156 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.628160 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.628164 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.628169 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.628173 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.628177 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.628181 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.628185 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.628189 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.628193 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.628198 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.628202 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.628206 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.628210 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.628214 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.628218 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.628222 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.628226 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.628231 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.628236 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.628241 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.628245 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.628249 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.628253 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.628257 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.628261 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.628266 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.628270 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:23 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.628274 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.628285 | controller | 16:16:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.628289 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:25 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.628294 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.628299 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.628304 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.628310 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.628315 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.628320 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.628325 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.628329 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.628333 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.628338 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.628342 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.628346 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.628350 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.628354 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.628360 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.628365 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.628370 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.628374 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.628378 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.628385 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.628390 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.628394 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.628398 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.628402 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.628406 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.628411 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.628415 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.628419 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.628423 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.628427 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.628431 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.628435 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.628440 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.628444 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.628448 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.628452 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.628456 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:25 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.628460 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.628465 | controller | 16:16:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.628475 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:26 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.628479 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.628484 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.628488 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.628492 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.628496 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.628500 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.628505 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.628509 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.628513 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.628517 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.628521 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.628525 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.628531 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.628535 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.628539 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.628543 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.628547 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.628552 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.628556 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.628560 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.628564 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.628568 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.628572 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.628577 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.628581 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.628585 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.628589 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.628593 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.628597 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.628602 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.628606 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.628610 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.628614 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.628618 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.628622 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.628627 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.628631 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:26 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.628635 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.628639 | controller | 16:16:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.628645 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:27 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.628649 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.628660 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.628664 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.628670 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.628674 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.628678 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.628682 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.628686 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.628691 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.628707 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.628712 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.628716 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.628720 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.628724 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.628730 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.628734 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.628738 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.628743 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.628747 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.628751 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.628755 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.628759 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.628763 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.628768 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.628772 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.628776 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.628780 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.628784 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.628788 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.628792 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.628796 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.628801 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.628805 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.628809 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.628813 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.628817 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.628823 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:27 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.628827 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.628832 | controller | 16:16:27 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.628836 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:28 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.628840 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.628844 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.628855 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.628860 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.628864 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.628868 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.628872 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.628877 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.628881 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.628885 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.628889 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.628893 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.628897 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.628903 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.628907 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.628912 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.628916 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.628920 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.628924 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.628928 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.628932 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.628937 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.628941 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.628945 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.628949 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.628953 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.628957 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.628962 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.628967 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.628971 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.628976 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.628980 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.628984 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.628988 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.628992 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.628996 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.629001 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:28 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.629005 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.629009 | controller | 16:16:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.629013 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:29 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.629017 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.629021 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.629026 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.629030 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.629040 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.629045 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.629049 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.629055 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.629061 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.629066 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.629072 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.629078 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.629083 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.629088 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.629092 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.629096 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.629100 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.629104 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.629108 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.629116 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.629120 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.629124 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.629128 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.629132 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.629136 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.629141 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.629145 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.629149 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.629153 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.629159 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.629164 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.629168 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.629173 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.629179 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.629185 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.629191 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.629196 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:29 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.629204 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.629209 | controller | 16:16:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.629213 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:30 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.629218 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.629222 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.629226 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.629230 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.629234 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.629246 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.629251 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.629255 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.629259 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.629263 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.629269 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.629275 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.629283 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.629289 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.629295 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.629300 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.629304 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.629308 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.629312 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.629319 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.629323 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.629327 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.629331 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.629336 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.629341 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.629346 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.629352 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.629358 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.629363 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.629369 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.629373 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.629378 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.629382 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.629386 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.629390 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.629394 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.629399 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:30 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.629403 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.629407 | controller | 16:16:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.629411 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:32 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.629415 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.629420 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.629424 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.629430 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.629434 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.629439 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.629451 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.629457 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.629463 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.629468 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.629473 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.629479 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.629487 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.629491 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.629495 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.629499 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.629504 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.629508 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.629512 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.629516 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.629520 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.629524 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.629529 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.629533 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.629539 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.629544 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.629550 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.629555 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.629560 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.629564 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.629568 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.629572 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.629576 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.629580 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.629584 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.629589 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.629596 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:32 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.629601 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.629607 | controller | 16:16:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.629612 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:33 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.629618 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.629622 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.629626 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.629633 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.629639 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.629644 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.629650 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.629655 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.629667 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.629673 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.629677 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.629681 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.629685 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.629689 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.629708 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.629712 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.629716 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.629721 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.629726 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.629732 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.629738 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.629743 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.629749 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.629755 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.629759 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.629763 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.629768 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.629773 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.629782 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.629788 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.629794 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.629798 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.629802 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.629806 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.629812 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.629817 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.629821 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:33 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.629827 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.629833 | controller | 16:16:33 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.629839 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:34 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.629844 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.629850 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.629854 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.629858 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.629862 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.629866 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.629871 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.629875 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.629879 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.629892 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.629897 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.629901 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.629905 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.629909 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.629914 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.629918 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.629922 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.629926 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.629931 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.629938 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.629942 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.629946 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.629951 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.629955 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.629959 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.629963 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.629967 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.629971 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.629976 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.629981 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.629987 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.629992 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.629998 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.630003 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.630007 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.630011 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.630015 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:34 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.630019 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.630023 | controller | 16:16:34 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.630027 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:35 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.630032 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.630036 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.630040 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.630044 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.630048 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.630052 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.630057 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.630061 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.630065 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.630069 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.630073 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.630088 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.630095 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.630099 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.630105 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.630109 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.630113 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.630117 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.630122 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.630126 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.630130 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.630134 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.630138 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.630143 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.630147 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.630151 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.630155 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.630159 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.630164 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.630168 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.630172 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.630176 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.630180 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.630184 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.630189 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.630193 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.630197 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:35 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.630201 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.630205 | controller | 16:16:35 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.630209 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:36 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.630214 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.630218 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.630222 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.630229 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.630235 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.630239 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.630243 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.630247 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.630252 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.630256 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.630260 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.630264 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.630276 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.630281 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.630285 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.630289 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.630294 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.630298 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.630302 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.630306 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.630312 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.630316 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.630321 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.630325 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.630329 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.630333 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.630337 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.630341 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.630346 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.630350 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.630354 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.630358 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.630364 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.630368 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.630372 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.630377 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.630382 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:36 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.630386 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.630391 | controller | 16:16:36 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.630395 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:37 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.630399 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.630403 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.630407 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.630412 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.630416 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.630420 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.630424 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.630428 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.630433 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.630437 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.630441 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.630445 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.630449 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.630453 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.630465 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.630469 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.630474 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.630478 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.630482 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.630486 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.630491 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.630495 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.630499 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.630503 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.630507 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.630511 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.630515 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.630520 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.630525 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.630531 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.630535 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.630539 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.630543 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.630548 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.630552 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.630556 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.630560 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:37 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.630564 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.630569 | controller | 16:16:37 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.630573 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:38 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.630577 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.630581 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.630585 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.630589 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.630594 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.630598 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.630602 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.630606 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.630610 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.630615 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.630620 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.630624 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.630629 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.630633 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.630637 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.630648 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.630652 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.630656 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.630660 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.630666 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.630670 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.630675 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.630679 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.630683 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.630687 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.630702 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.630709 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.630713 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.630718 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.630722 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.630726 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.630730 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.630734 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.630739 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.630743 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.630747 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.630751 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:38 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.630755 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.630759 | controller | 16:16:39 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.630764 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:40 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.630768 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.630772 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.630776 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.630780 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.630784 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.630789 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.630793 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.630797 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.630801 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.630805 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.630810 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.630814 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.630819 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.630824 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.630828 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.630832 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.630843 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.630848 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.630852 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.630856 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.630861 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.630865 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.630869 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.630873 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.630877 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.630881 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.630886 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.630892 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.630896 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.630900 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.630904 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.630909 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.630913 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.630919 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.630925 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.630932 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.630938 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:40 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.630944 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.630949 | controller | 16:16:40 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.630955 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:41 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.630961 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.630965 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.630970 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.630978 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.630982 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.630987 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.630991 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.630995 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.630999 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.631004 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.631008 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.631012 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.631016 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.631020 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.631025 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.631029 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.631033 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.631037 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.631050 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.631055 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.631059 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.631064 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.631068 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.631072 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.631076 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.631081 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.631085 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.631089 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.631093 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.631098 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.631102 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.631106 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.631110 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.631115 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.631119 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.631123 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.631133 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:41 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.631137 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.631141 | controller | 16:16:41 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.631146 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:42 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.631150 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.631154 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.631160 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.631165 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.631171 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.631177 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.631182 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.631187 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.631191 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.631197 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.631202 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.631206 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.631210 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.631214 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.631218 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.631223 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.631227 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.631231 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.631235 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.631250 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.631256 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.631260 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.631265 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.631269 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.631273 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.631277 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.631282 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.631286 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.631293 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.631298 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.631303 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.631307 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.631313 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.631318 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.631322 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.631327 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.631331 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:42 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.631335 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.631339 | controller | 16:16:42 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.631343 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:43 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.631348 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.631352 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.631356 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.631360 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.631364 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.631368 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.631373 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.631377 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.631381 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.631385 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.631389 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.631394 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.631398 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.631402 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.631408 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.631412 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.631416 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.631420 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.631425 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.631430 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.631442 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.631447 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.631451 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.631455 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.631459 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.631463 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.631470 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.631474 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.631478 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.631482 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.631487 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.631491 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.631495 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.631499 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.631504 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.631508 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.631512 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:43 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.631516 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.631520 | controller | 16:16:43 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.631540 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:44 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.631545 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.631550 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.631555 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.631560 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.631565 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.631570 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.631575 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.631580 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.631585 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.631589 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.631594 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.631599 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.631606 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.631611 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.631616 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.631621 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.631626 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.631631 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.631636 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.631641 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.631646 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.631659 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.631664 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.631669 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.631674 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.631679 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.631684 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.631689 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.631705 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.631713 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.631718 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.631723 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.631728 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.631733 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.631738 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.631743 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.631748 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:44 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.631752 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.631757 | controller | 16:16:44 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.631762 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:45 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.631769 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.631774 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.631779 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.631787 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.631791 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.631795 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.631800 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.631804 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.631809 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.631814 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.631818 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.631822 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.631826 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.631830 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.631835 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.631839 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.631843 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.631847 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.631851 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.631855 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.631860 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.631864 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.631868 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.631880 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.631885 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.631889 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.631893 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.631897 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.631901 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.631908 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.631912 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.631916 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.631920 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.631924 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.631928 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.631933 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.631938 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:45 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.631942 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.631946 | controller | 16:16:46 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.631951 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:47 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.631955 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.631959 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.631963 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.631967 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.631972 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.631976 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.631980 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.631984 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.631988 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.631992 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.631997 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.632001 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.632005 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.632009 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.632013 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.632018 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.632022 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.632026 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.632030 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.632034 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.632039 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.632043 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.632047 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.632051 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.632057 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.632067 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.632072 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.632076 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.632082 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.632086 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.632090 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.632094 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.632099 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.632103 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.632107 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.632111 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.632115 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:47 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.632120 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.632124 | controller | 16:16:47 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.632128 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:48 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.632132 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.632136 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.632140 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.632145 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.632149 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.632153 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.632157 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.632161 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.632165 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.632170 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.632174 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.632178 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.632182 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.632186 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.632190 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.632195 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.632199 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.632203 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.632207 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.632214 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.632218 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.632223 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.632227 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.632231 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.632235 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.632239 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.632249 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.632254 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.632258 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.632262 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.632267 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.632271 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.632275 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.632279 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.632283 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.632287 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.632292 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:48 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.632296 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.632300 | controller | 16:16:48 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.632304 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:49 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.632308 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.632313 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.632317 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.632324 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.632328 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.632332 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.632336 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.632342 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.632346 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.632350 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.632354 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.632359 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.632364 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.632368 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.632373 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.632377 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.632381 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.632385 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.632389 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.632394 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.632398 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.632402 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.632406 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.632410 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.632415 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.632419 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.632423 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.632434 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.632438 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.632442 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.632447 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.632451 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.632455 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.632460 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.632464 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.632469 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.632475 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:49 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.632482 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.632488 | controller | 16:16:49 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.632494 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:50 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.632500 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.632506 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.632511 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.632518 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.632522 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.632527 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.632531 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.632535 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.632539 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.632543 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.632547 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.632552 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.632556 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.632560 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.632564 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.632568 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.632573 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.632577 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.632581 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.632585 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.632589 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.632593 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.632598 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.632603 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.632608 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.632612 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.632616 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.632620 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.632624 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.632637 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.632642 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.632646 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.632650 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.632655 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.632659 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.632663 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.632669 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:50 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.632673 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.632677 | controller | 16:16:50 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.632681 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:51 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.632686 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.632690 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.632708 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.632712 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.632716 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.632720 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.632724 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.632729 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.632733 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.632737 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.632741 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.632745 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.632749 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.632754 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.632759 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.632764 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.632768 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.632772 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.632776 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.632780 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.632785 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.632789 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.632793 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.632797 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.632801 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.632806 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.632810 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.632814 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.632820 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.632824 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.632835 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.632840 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.632844 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.632848 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.632852 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.632857 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.632861 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:51 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.632865 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.632869 | controller | 16:16:51 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.632873 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:52 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.632878 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.632882 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.632887 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.632891 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.632896 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.632900 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.632904 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.632908 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.632912 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.632917 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.632921 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.632925 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.632929 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.632933 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.632937 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.632942 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.632946 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.632951 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.632957 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.632964 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.632970 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.632976 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.632981 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.632985 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.632989 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.632993 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.632997 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.633002 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.633006 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.633010 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.633014 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.633026 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.633031 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.633035 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.633039 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.633044 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.633048 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:52 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.633054 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.633059 | controller | 16:16:53 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.633065 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:54 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.633071 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.633077 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.633082 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.633089 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.633094 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.633100 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.633106 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.633112 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.633118 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.633124 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.633129 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.633135 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.633144 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.633150 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.633155 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.633161 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.633167 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.633173 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.633178 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.633185 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.633190 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.633194 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.633200 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.633204 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.633210 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.633216 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.633222 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.633228 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.633234 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.633242 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.633248 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.633254 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.633259 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.633273 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.633278 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.633283 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.633287 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:54 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.633292 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.633299 | controller | 16:16:54 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.633304 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:55 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.633308 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.633312 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.633316 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.633323 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.633328 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.633332 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.633336 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.633340 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.633345 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.633349 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.633353 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.633357 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.633362 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.633366 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.633371 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.633377 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.633383 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.633388 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.633394 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.633398 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.633402 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.633406 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.633411 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.633415 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.633421 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.633426 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.633432 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.633437 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.633443 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.633447 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.633451 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.633456 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.633460 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.633464 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.633478 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.633482 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.633489 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:55 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.633494 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.633498 | controller | 16:16:55 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.633502 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:56 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.633507 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.633513 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.633517 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.633521 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.633526 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.633531 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.633536 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.633542 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.633548 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.633554 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.633560 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.633564 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.633569 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.633573 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.633577 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.633581 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.633586 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.633590 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.633594 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.633598 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.633603 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.633607 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.633612 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.633617 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.633623 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.633629 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.633635 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.633640 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.633648 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.633654 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.633658 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.633662 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.633667 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.633671 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.633675 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.633688 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.633705 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:56 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.633713 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.633718 | controller | 16:16:56 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.633723 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:57 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.633727 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.633731 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.633736 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.633744 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.633751 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.633757 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.633763 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.633769 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.633775 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.633781 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.633787 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.633793 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.633799 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.633804 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.633810 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.633816 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.633821 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.633825 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.633829 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.633836 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.633843 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.633848 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.633852 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.633856 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.633860 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.633865 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.633869 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.633873 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.633877 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.633881 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.633886 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.633890 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.633894 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.633898 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.633903 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.633907 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.633911 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:57 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.633925 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.633930 | controller | 16:16:57 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.633934 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:58 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.633939 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.633943 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.633947 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.633951 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.633956 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.633962 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.633968 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.633973 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.633979 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.633985 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.633991 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.633996 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.634003 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.634008 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.634012 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.634016 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.634020 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.634025 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.634029 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.634034 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.634039 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.634045 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.634050 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.634056 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.634062 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.634068 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.634074 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.634079 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.634085 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.634090 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.634094 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.634098 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.634103 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.634109 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.634115 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.634120 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.634126 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:58 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.634132 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.634146 | controller | 16:16:58 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.634153 | controller | \"ac-watcher\" not found\n logger.go:42: 16:16:59 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.634162 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.634168 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.634172 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.634178 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.634182 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.634186 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.634191 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.634195 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.634199 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.634203 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.634208 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.634212 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.634216 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.634220 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.634226 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.634230 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.634235 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.634239 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.634243 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.634249 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.634254 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.634258 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.634262 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.634267 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.634271 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.634275 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.634279 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.634284 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.634289 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.634295 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.634300 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.634306 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.634312 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.634317 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.634322 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.634326 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.634332 | controller | expected application_credential_id\"]\n logger.go:42: 16:16:59 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.634336 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.634340 | controller | 16:17:00 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.634353 | controller | \"ac-watcher\" not found\n logger.go:42: 16:17:01 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.634358 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.634362 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.634367 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.634371 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.634375 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.634379 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.634384 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.634388 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.634392 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.634396 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.634401 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.634405 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.634409 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.634413 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.634417 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.634423 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.634428 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.634432 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.634436 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.634440 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.634444 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.634449 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.634453 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.634457 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.634461 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.634465 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.634470 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.634474 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.634481 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.634486 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.634490 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.634494 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.634499 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.634503 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.634507 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.634511 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.634515 | controller | expected application_credential_id\"]\n logger.go:42: 16:17:01 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.634520 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.634524 | controller | 16:17:01 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.634528 | controller | \"ac-watcher\" not found\n logger.go:42: 16:17:02 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.634532 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.634544 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.634548 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.634553 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.634558 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.634562 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.634566 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.634570 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.634575 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.634579 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.634583 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.634587 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.634592 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.634596 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.634600 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.634604 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.634608 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.634612 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.634617 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.634622 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.634627 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.634631 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.634635 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.634639 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.634643 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.634647 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.634652 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.634656 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.634660 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.634666 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.634670 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.634674 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.634678 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.634683 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.634687 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.634704 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.634711 | controller | expected application_credential_id\"]\n logger.go:42: 16:17:02 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.634715 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.634720 | controller | 16:17:02 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.634726 | controller | \"ac-watcher\" not found\n logger.go:42: 16:17:03 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.634730 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.634734 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.634746 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.634751 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.634758 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.634764 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.634770 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.634776 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.634781 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.634787 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.634792 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.634797 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.634804 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.634808 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.634812 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.634817 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.634823 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.634829 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.634834 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.634840 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.634844 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.634849 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.634853 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.634857 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.634861 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.634865 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.634870 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.634874 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.634878 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.634882 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.634886 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.634891 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.634895 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.634900 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.634905 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.634911 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.634916 | controller | expected application_credential_id\"]\n logger.go:42: 16:17:03 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.634922 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.634926 | controller | 16:17:03 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.634930 | controller | \"ac-watcher\" not found\n logger.go:42: 16:17:04 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.634934 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.634938 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.634942 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.634949 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.634962 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.634966 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.634970 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.634975 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.634980 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.634986 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.634991 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.634997 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.635002 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.635009 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.635014 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.635020 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.635026 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.635030 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.635034 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.635038 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.635042 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.635047 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.635051 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.635055 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.635059 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.635063 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.635067 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.635071 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.635076 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.635080 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.635086 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.635091 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.635097 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.635103 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.635109 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.635114 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.635121 | controller | expected application_credential_id\"]\n logger.go:42: 16:17:04 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.635125 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.635129 | controller | 16:17:04 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.635134 | controller | \"ac-watcher\" not found\n logger.go:42: 16:17:05 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.635138 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.635142 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.635146 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.635153 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.635158 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.635171 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.635175 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.635180 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.635185 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.635189 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.635193 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.635197 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.635201 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.635205 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.635210 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.635214 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.635218 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.635222 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.635226 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.635232 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.635237 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.635241 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.635245 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.635249 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.635253 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.635257 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.635262 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.635266 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.635272 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.635276 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.635280 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.635284 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.635288 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.635292 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.635297 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.635301 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.635305 | controller | expected application_credential_id\"]\n logger.go:42: 16:17:05 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.635311 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.635315 | controller | 16:17:05 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.635319 | controller | \"ac-watcher\" not found\n logger.go:42: 16:17:06 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.635323 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.635327 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.635332 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.635336 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.635340 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.635344 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.635356 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.635360 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.635364 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.635369 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.635373 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.635377 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.635381 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.635385 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.635389 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.635394 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.635398 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.635402 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.635406 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.635413 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.635417 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.635421 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.635425 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.635430 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.635434 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.635438 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.635442 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.635446 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.635450 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.635455 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.635459 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.635463 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.635467 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.635471 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.635475 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.635480 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.635484 | controller | expected application_credential_id\"]\n logger.go:42: 16:17:06 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.635488 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.635492 | controller | 16:17:07 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.635496 | controller | \"ac-watcher\" not found\n logger.go:42: 16:17:08 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.635501 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.635505 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.635509 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.635513 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.635517 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.635522 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.635526 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.635544 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.635557 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.635563 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.635568 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.635573 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.635583 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.635588 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.635598 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.635603 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.635608 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.635613 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.635618 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.635623 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.635628 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.635633 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.635638 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.635642 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.635647 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.635652 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.635657 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.635662 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.635667 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.635672 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.635677 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.635681 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.635686 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.635691 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.635711 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.635717 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.635722 | controller | expected application_credential_id\"]\n logger.go:42: 16:17:08 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.635727 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.635732 | controller | 16:17:08 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.635736 | controller | \"ac-watcher\" not found\n logger.go:42: 16:17:09 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.635741 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.635746 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.635751 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.635758 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.635763 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.635768 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.635773 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.635777 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.635783 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.635795 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.635800 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.635804 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.635808 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.635812 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.635817 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.635821 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.635825 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.635829 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.635833 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.635838 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.635842 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.635846 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.635850 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.635854 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.635858 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.635863 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.635867 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.635871 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.635875 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.635879 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.635885 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.635890 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.635894 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.635898 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.635904 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.635908 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.635915 | controller | expected application_credential_id\"]\n logger.go:42: 16:17:09 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.635919 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.635923 | controller | 16:17:09 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.635927 | controller | \"ac-watcher\" not found\n logger.go:42: 16:17:10 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.635931 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.635936 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.635940 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.635944 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.635948 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.635952 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.635957 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.635961 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.635965 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.635969 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.635973 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.635985 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.635989 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.635994 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.635998 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.636003 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.636007 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.636011 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.636015 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.636019 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.636024 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.636028 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.636032 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.636036 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.636040 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.636044 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.636048 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.636053 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.636058 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.636069 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.636074 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.636078 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.636082 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.636086 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.636091 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.636095 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.636099 | controller | expected application_credential_id\"]\n logger.go:42: 16:17:10 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.636103 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.636107 | controller | 16:17:10 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.636112 | controller | \"ac-watcher\" not found\n logger.go:42: 16:17:11 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.636116 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.636120 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.636124 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.636128 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.636133 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.636137 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.636141 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.636145 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.636149 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.636154 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.636158 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.636163 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.636175 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.636180 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.636184 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.636188 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.636193 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.636197 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.636202 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.636212 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.636218 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.636224 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.636230 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.636235 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.636239 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.636243 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.636249 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.636254 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.636260 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.636265 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.636271 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.636276 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.636280 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.636284 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.636288 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.636292 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.636296 | controller | expected application_credential_id\"]\n logger.go:42: 16:17:11 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.636301 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.636305 | controller | 16:17:11 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.636309 | controller | \"ac-watcher\" not found\n logger.go:42: 16:17:12 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.636313 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.636317 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.636321 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.636326 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.636330 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.636334 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.636338 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.636342 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.636347 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.636351 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.636355 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.636359 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.636365 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.636370 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.636383 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.636388 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.636394 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.636399 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.636405 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.636410 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.636416 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.636420 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.636424 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.636429 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.636433 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.636438 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.636444 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.636449 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.636455 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.636459 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.636463 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.636470 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.636474 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.636479 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.636485 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.636490 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.636496 | controller | expected application_credential_id\"]\n logger.go:42: 16:17:12 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.636501 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.636506 | controller | 16:17:12 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.636512 | controller | \"ac-watcher\" not found\n logger.go:42: 16:17:13 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.636517 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.636521 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.636525 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.636533 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.636538 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.636542 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.636546 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.636550 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.636554 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.636559 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.636565 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.636570 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.636576 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.636582 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.636588 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.636601 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.636606 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.636610 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.636614 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.636618 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.636623 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.636628 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.636634 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.636639 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.636645 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.636650 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.636656 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.636660 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.636665 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.636669 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.636673 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.636677 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.636681 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.636686 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.636690 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.636710 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.636718 | controller | expected application_credential_id\"]\n logger.go:42: 16:17:13 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.636723 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.636727 | controller | 16:17:14 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.636731 | controller | \"ac-watcher\" not found\n logger.go:42: 16:17:15 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.636735 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.636739 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.636744 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.636748 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.636752 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.636756 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.636761 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.636765 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.636769 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.636773 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.636779 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.636784 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.636788 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.636792 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.636796 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.636800 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.636814 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.636818 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.636823 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.636827 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.636831 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.636835 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.636840 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.636844 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.636848 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.636852 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.636856 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.636860 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.636867 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.636872 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.636876 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.636880 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.636884 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.636888 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.636893 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.636897 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.636901 | controller | expected application_credential_id\"]\n logger.go:42: 16:17:15 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.636905 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.636909 | controller | 16:17:15 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.636913 | controller | \"ac-watcher\" not found\n logger.go:42: 16:17:16 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.636918 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.636922 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.636926 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.636930 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.636934 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.636939 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.636943 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.636947 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.636951 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.636955 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.636959 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.636964 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.636968 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.636972 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.636978 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.636982 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.636986 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.636998 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.637003 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.637009 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.637013 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.637017 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.637021 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.637026 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.637030 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.637034 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.637038 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.637044 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.637048 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.637052 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.637056 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.637061 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.637065 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.637069 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.637073 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.637077 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.637082 | controller | expected application_credential_id\"]\n logger.go:42: 16:17:16 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.637087 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.637092 | controller | 16:17:16 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.637096 | controller | \"ac-watcher\" not found\n logger.go:42: 16:17:17 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.637100 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.637104 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.637109 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.637113 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.637117 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.637121 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.637125 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.637129 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.637134 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.637138 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.637142 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.637146 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.637152 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.637156 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.637160 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.637164 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.637168 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.637172 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.637177 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.637190 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.637195 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.637199 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.637203 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.637207 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.637211 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.637216 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.637220 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.637224 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.637228 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.637232 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.637236 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.637240 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.637245 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.637249 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.637253 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.637257 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.637261 | controller | expected application_credential_id\"]\n logger.go:42: 16:17:17 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.637266 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.637270 | controller | 16:17:17 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.637274 | controller | \"ac-watcher\" not found\n logger.go:42: 16:17:18 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.637278 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.637282 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.637287 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.637293 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.637298 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.637304 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.637308 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.637313 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.637318 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.637325 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.637329 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.637333 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.637337 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.637341 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.637346 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.637350 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.637354 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.637358 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.637362 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.637367 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.637378 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.637383 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.637387 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.637391 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.637396 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.637400 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.637404 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.637408 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.637412 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.637419 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.637423 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.637427 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.637431 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.637436 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.637440 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.637444 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.637450 | controller | expected application_credential_id\"]\n logger.go:42: 16:17:18 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.637454 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.637458 | controller | 16:17:18 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.637462 | controller | \"ac-watcher\" not found\n logger.go:42: 16:17:19 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.637467 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.637471 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.637475 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.637479 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.637483 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.637487 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.637492 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.637496 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.637500 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.637504 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.637508 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.637513 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.637517 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.637521 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.637525 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.637529 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.637533 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.637538 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.637542 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.637546 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.637550 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.637563 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.637568 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.637572 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.637576 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.637581 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.637586 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.637591 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.637596 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.637600 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.637605 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.637609 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.637613 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.637617 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.637621 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.637625 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.637630 | controller | expected application_credential_id\"]\n logger.go:42: 16:17:19 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.637634 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.637638 | controller | 16:17:19 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.637642 | controller | \"ac-watcher\" not found\n logger.go:42: 16:17:20 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.637646 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.637650 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.637655 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.637659 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.637663 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.637667 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.637671 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.637675 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.637679 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.637684 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.637688 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.637703 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.637710 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.637714 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.637718 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.637722 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.637727 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.637731 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.637735 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.637742 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.637746 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.637751 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.637755 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.637767 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.637772 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.637776 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.637780 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.637784 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.637788 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.637793 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.637797 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.637801 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.637805 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.637809 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.637814 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.637818 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.637822 | controller | expected application_credential_id\"]\n logger.go:42: 16:17:20 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.637826 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.637830 | controller | 16:17:21 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.637834 | controller | \"ac-watcher\" not found\n logger.go:42: 16:17:22 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.637839 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.637843 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.637847 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.637853 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.637857 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.637862 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.637866 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.637870 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.637876 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.637880 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.637884 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.637888 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.637894 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.637898 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.637902 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.637906 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.637910 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.637915 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.637919 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.637923 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.637927 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.637931 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.637936 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.637940 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.637951 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.637955 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.637960 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.637964 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.637968 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.637972 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.637976 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.637981 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.637985 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.637989 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.637993 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.637997 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.638002 | controller | expected application_credential_id\"]\n logger.go:42: 16:17:22 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.638006 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.638010 | controller | 16:17:22 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.638014 | controller | \"ac-watcher\" not found\n logger.go:42: 16:17:23 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.638019 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.638023 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.638027 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.638034 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.638038 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.638042 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.638046 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.638051 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.638055 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.638059 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.638063 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.638067 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.638071 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.638076 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.638080 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.638084 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.638088 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.638092 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.638097 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.638102 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.638107 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.638111 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.638115 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.638119 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.638124 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.638129 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.638139 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.638144 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.638149 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.638153 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.638157 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.638161 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.638165 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.638170 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.638174 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.638178 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.638184 | controller | expected application_credential_id\"]\n logger.go:42: 16:17:23 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.638188 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.638192 | controller | 16:17:23 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.638197 | controller | \"ac-watcher\" not found\n logger.go:42: 16:17:24 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.638201 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.638205 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.638209 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.638213 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.638217 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.638222 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.638226 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.638230 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.638234 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.638238 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.638243 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.638247 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.638251 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.638255 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.638261 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.638265 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.638269 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.638273 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.638278 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.638282 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.638286 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.638290 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.638296 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.638300 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.638304 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.638308 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.638313 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.638324 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.638330 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.638335 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.638339 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.638343 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.638347 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.638351 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.638356 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.638360 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.638364 | controller | expected application_credential_id\"]\n logger.go:42: 16:17:24 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.638368 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.638372 | controller | 16:17:24 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.638376 | controller | \"ac-watcher\" not found\n logger.go:42: 16:17:25 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.638381 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.638385 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.638389 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.638393 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.638398 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.638402 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.638406 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.638412 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.638416 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.638420 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.638424 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.638429 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.638433 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.638437 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.638441 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.638446 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.638450 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.638454 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.638458 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.638465 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.638470 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.638474 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.638478 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.638482 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.638487 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.638491 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.638495 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.638499 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.638504 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.638514 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.638519 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.638523 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.638528 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.638532 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.638536 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.638540 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.638545 | controller | expected application_credential_id\"]\n logger.go:42: 16:17:25 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.638549 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.638553 | controller | 16:17:25 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.638557 | controller | \"ac-watcher\" not found\n logger.go:42: 16:17:26 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.638562 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.638566 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.638570 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.638574 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.638579 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.638583 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.638587 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.638591 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.638595 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.638600 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.638604 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.638608 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.638615 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.638619 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.638623 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.638628 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.638632 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.638636 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.638640 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.638645 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.638649 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.638653 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.638657 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.638663 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.638668 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.638672 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.638676 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.638680 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.638685 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.638702 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.638715 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.638720 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.638724 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.638729 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.638733 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.638737 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.638741 | controller | expected application_credential_id\"]\n logger.go:42: 16:17:26 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.638746 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.638750 | controller | 16:17:26 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.638754 | controller | \"ac-watcher\" not found\n logger.go:42: 16:17:27 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.638759 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.638763 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.638767 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.638773 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.638777 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.638781 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.638786 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.638790 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.638794 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.638799 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.638803 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.638807 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.638811 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.638815 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.638820 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.638824 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.638828 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.638832 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.638837 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.638841 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.638845 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.638850 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.638854 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.638858 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.638862 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.638867 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.638871 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.638875 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.638879 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.638883 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.638888 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.638897 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.638902 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.638906 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.638911 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.638915 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.638922 | controller | expected application_credential_id\"]\n logger.go:42: 16:17:27 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.638926 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.638930 | controller | 16:17:28 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.638935 | controller | \"ac-watcher\" not found\n logger.go:42: 16:17:29 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.638939 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.638943 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.638949 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.638953 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.638958 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.638962 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.638966 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.638970 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.638975 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.638979 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.638983 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.638987 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.638992 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.638996 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.639000 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.639004 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.639009 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.639013 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.639017 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.639023 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.639027 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.639031 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.639036 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.639040 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.639044 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.639048 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.639053 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.639057 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.639063 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.639067 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.639071 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.639075 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.639086 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.639090 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.639095 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.639099 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.639103 | controller | expected application_credential_id\"]\n logger.go:42: 16:17:29 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.639108 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.639112 | controller | 16:17:29 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.639116 | controller | \"ac-watcher\" not found\n logger.go:42: 16:17:30 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.639120 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.639125 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.639129 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.639135 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.639141 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.639146 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.639150 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.639154 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.639158 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.639163 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.639167 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.639171 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.639175 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.639180 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.639184 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.639188 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.639192 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.639197 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.639201 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.639207 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.639211 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.639215 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.639219 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.639224 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.639228 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.639232 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.639238 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.639242 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.639246 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.639250 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.639255 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.639259 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.639263 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.639268 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.639278 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.639283 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.639287 | controller | expected application_credential_id\"]\n logger.go:42: 16:17:30 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.639292 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.639296 | controller | 16:17:30 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.639300 | controller | \"ac-watcher\" not found\n logger.go:42: 16:17:31 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.639305 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.639309 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.639313 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.639317 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.639322 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.639326 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.639330 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.639335 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.639339 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.639343 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.639347 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.639352 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.639357 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.639361 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.639366 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.639370 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.639374 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.639379 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.639383 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.639387 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.639391 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.639396 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.639400 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.639404 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.639408 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.639413 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.639417 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.639421 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.639425 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.639430 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.639434 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.639438 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.639442 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.639447 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.639451 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.639461 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.639467 | controller | expected application_credential_id\"]\n logger.go:42: 16:17:31 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.639471 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.639475 | controller | 16:17:31 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.639480 | controller | \"ac-watcher\" not found\n logger.go:42: 16:17:32 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.639484 | controller | | running command: [sh -c set -euo pipefail\n NS=\"${NAMESPACE}\"\n \n 2026-03-20 16:26:25.639490 | controller | \ echo \"Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\"\n 2026-03-20 16:26:25.639494 | controller | \ oc wait -n \"${NS}\" appcred/ac-watcher --for=condition=Ready --timeout=300s\n 2026-03-20 16:26:25.639500 | controller | \ \n ac_id=$(oc get -n \"${NS}\" appcred/ac-watcher -o jsonpath='{.status.acID}')\n 2026-03-20 16:26:25.639504 | controller | \ if [ -z \"${ac_id}\" ]; then\n echo \"ERROR: appcred/ac-watcher.status.acID 2026-03-20 16:26:25.639508 | controller | is empty\"\n exit 1\n fi\n echo \"ac-watcher.status.acID=${ac_id}\"\n 2026-03-20 16:26:25.639513 | controller | \ \n oc get -n \"${NS}\" secret/ac-watcher-secret >/dev/null\n secret_ac_id=$(oc 2026-03-20 16:26:25.639517 | controller | get -n \"${NS}\" secret/ac-watcher-secret -o jsonpath='{.data.AC_ID}' | base64 -d)\n 2026-03-20 16:26:25.639521 | controller | \ if [ \"${secret_ac_id}\" != \"${ac_id}\" ]; then\n echo \"ERROR: 2026-03-20 16:26:25.639525 | controller | Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})\"\n exit 2026-03-20 16:26:25.639531 | controller | 1\n fi\n echo \"✓ Secret AC_ID matches appcred.status.acID\"\n \n 2026-03-20 16:26:25.639556 | controller | \ echo \"Waiting for watcher rollouts...\"\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.639561 | controller | statefulset/watcher-kuttl-api --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.639566 | controller | statefulset/watcher-kuttl-applier --timeout=300s\n oc rollout status -n \"${NS}\" 2026-03-20 16:26:25.639573 | controller | statefulset/watcher-kuttl-decision-engine --timeout=300s\n \n old_api_uid=$(oc 2026-03-20 16:26:25.639579 | controller | get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.api_uid}')\n 2026-03-20 16:26:25.639584 | controller | \ old_applier_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.applier_uid}')\n 2026-03-20 16:26:25.639589 | controller | \ old_de_uid=$(oc get -n \"${NS}\" configmap/appcred-watcher-pre -o jsonpath='{.data.decision_engine_uid}')\n 2026-03-20 16:26:25.639594 | controller | \ \n new_api_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-api-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.639599 | controller | \ new_applier_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-applier-0 -o jsonpath='{.metadata.uid}')\n 2026-03-20 16:26:25.639604 | controller | \ new_de_uid=$(oc get pod -n \"${NS}\" watcher-kuttl-decision-engine-0 -o 2026-03-20 16:26:25.639609 | controller | jsonpath='{.metadata.uid}')\n \n if [ \"${old_api_uid}\" = \"${new_api_uid}\" 2026-03-20 16:26:25.639613 | controller | ] || [ \"${old_applier_uid}\" = \"${new_applier_uid}\" ] || [ \"${old_de_uid}\" 2026-03-20 16:26:25.639618 | controller | = \"${new_de_uid}\" ]; then\n echo \"ERROR: expected watcher pods to restart 2026-03-20 16:26:25.639623 | controller | after appcred secret became available\"\n echo \" api: ${old_api_uid} 2026-03-20 16:26:25.639628 | controller | -> ${new_api_uid}\"\n echo \" applier: ${old_applier_uid} -> ${new_applier_uid}\"\n 2026-03-20 16:26:25.639633 | controller | \ echo \" decision-engine: ${old_de_uid} -> ${new_de_uid}\"\n exit 2026-03-20 16:26:25.639638 | controller | 1\n fi\n echo \"✓ watcher pods restarted after appcred secret became 2026-03-20 16:26:25.639643 | controller | available\"\n \n echo \"Checking watcher config contains application_credential_id...\"\n 2026-03-20 16:26:25.639648 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-api-0 -c watcher-api -- \\\n bash 2026-03-20 16:26:25.639653 | controller | -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.639658 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-applier-0 -c watcher-applier -- \\\n 2026-03-20 16:26:25.639663 | controller | \ bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" /etc/watcher/watcher.conf.d/00-default.conf\"\n 2026-03-20 16:26:25.639668 | controller | \ oc exec -n \"${NS}\" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.639673 | controller | -- \\\n bash -c \"grep -q \\\"^application_credential_id = ${ac_id}$\\\" 2026-03-20 16:26:25.639678 | controller | /etc/watcher/watcher.conf.d/00-default.conf\"\n echo \"✓ watcher config contains 2026-03-20 16:26:25.639685 | controller | expected application_credential_id\"]\n logger.go:42: 16:17:32 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.639711 | controller | | Waiting for KeystoneApplicationCredential ac-watcher to be Ready...\n logger.go:42: 2026-03-20 16:26:25.639720 | controller | 16:17:32 | appcred-tests/2-deploy-appcred | Error from server (NotFound): keystoneapplicationcredentials.keystone.openstack.org 2026-03-20 16:26:25.639725 | controller | \"ac-watcher\" not found\n logger.go:42: 16:17:33 | appcred-tests/2-deploy-appcred 2026-03-20 16:26:25.639730 | controller | | test step failed 2-deploy-appcred\n case.go:396: failed in step 2-deploy-appcred\n 2026-03-20 16:26:25.639735 | controller | \ case.go:398: command \"NS=\\\"${NAMESPACE}\\\"\\\\n echo \\\"Waiting for KeystoneApplicationCredential...\" 2026-03-20 16:26:25.639740 | controller | failed, exit status 1\n logger.go:42: 16:17:33 | appcred-tests | skipping kubernetes 2026-03-20 16:26:25.639745 | controller | event logging\n=== CONT kuttl/harness/watcher-notification\n logger.go:42: 16:17:33 2026-03-20 16:26:25.639750 | controller | | watcher-notification | Skipping creation of user-supplied namespace: watcher-kuttl-default\n 2026-03-20 16:26:25.639755 | controller | \ logger.go:42: 16:17:33 | watcher-notification/0-cleanup-watcher | starting test 2026-03-20 16:26:25.639760 | controller | step 0-cleanup-watcher\n logger.go:42: 16:17:33 | watcher-notification/0-cleanup-watcher 2026-03-20 16:26:25.639765 | controller | | test step completed 0-cleanup-watcher\n logger.go:42: 16:17:33 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.639770 | controller | | starting test step 1-deploy-with-notification\n logger.go:42: 16:17:33 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.639775 | controller | | Watcher:watcher-kuttl-default/watcher-kuttl created\n logger.go:42: 16:17:33 2026-03-20 16:26:25.639780 | controller | | watcher-notification/1-deploy-with-notification | running command: [sh -c set 2026-03-20 16:26:25.639784 | controller | -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get 2026-03-20 16:26:25.639790 | controller | pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep 2026-03-20 16:26:25.639794 | controller | -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh 2026-03-20 16:26:25.639798 | controller | -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep 2026-03-20 16:26:25.639802 | controller | -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-20 16:26:25.639807 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-20 16:26:25.639811 | controller | 16:17:33 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-20 16:26:25.639815 | controller | \ logger.go:42: 16:17:34 | watcher-notification/1-deploy-with-notification | Now 2026-03-20 16:26:25.639823 | controller | using project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.639827 | controller | \ logger.go:42: 16:17:34 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.639832 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.639836 | controller | \ logger.go:42: 16:17:34 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.639840 | controller | head -1\n logger.go:42: 16:17:34 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.639844 | controller | | ++ grep -v '^$'\n logger.go:42: 16:17:34 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.639848 | controller | | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 16:17:34 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.639853 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 16:17:34 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.639859 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-20 16:26:25.639863 | controller | \ logger.go:42: 16:17:34 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.639867 | controller | grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n 2026-03-20 16:26:25.639871 | controller | \ logger.go:42: 16:17:35 | watcher-notification/1-deploy-with-notification | error: 2026-03-20 16:26:25.639876 | controller | Internal error occurred: error executing command in container: container is not 2026-03-20 16:26:25.639880 | controller | created or running\n logger.go:42: 16:17:35 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.639884 | controller | | ++ echo\n logger.go:42: 16:17:35 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.639888 | controller | | + '[' 0 == 1 ']'\n logger.go:42: 16:17:36 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.639892 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-20 16:26:25.639897 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-20 16:26:25.639901 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-20 16:26:25.639913 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.639918 | controller | |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-20 16:26:25.639922 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-20 16:26:25.639926 | controller | 16:17:36 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-20 16:26:25.639931 | controller | \ logger.go:42: 16:17:36 | watcher-notification/1-deploy-with-notification | Already 2026-03-20 16:26:25.639935 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.639939 | controller | \ logger.go:42: 16:17:36 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.639943 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.639948 | controller | \ logger.go:42: 16:17:36 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.639952 | controller | head -1\n logger.go:42: 16:17:36 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.639956 | controller | | ++ grep -v '^$'\n logger.go:42: 16:17:36 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.639961 | controller | | + APIPOD=\n logger.go:42: 16:17:37 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.639965 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-20 16:26:25.639969 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-20 16:26:25.639973 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-20 16:26:25.639977 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.639982 | controller | |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-20 16:26:25.639986 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-20 16:26:25.639990 | controller | 16:17:37 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-20 16:26:25.639994 | controller | \ logger.go:42: 16:17:37 | watcher-notification/1-deploy-with-notification | Already 2026-03-20 16:26:25.639999 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.640003 | controller | \ logger.go:42: 16:17:37 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.640008 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.640013 | controller | \ logger.go:42: 16:17:37 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.640017 | controller | head -1\n logger.go:42: 16:17:37 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.640021 | controller | | ++ grep -v '^$'\n logger.go:42: 16:17:37 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.640028 | controller | | + APIPOD=\n logger.go:42: 16:17:39 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.640032 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-20 16:26:25.640037 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-20 16:26:25.640041 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-20 16:26:25.640045 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.640050 | controller | |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-20 16:26:25.640054 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-20 16:26:25.640058 | controller | 16:17:39 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-20 16:26:25.640062 | controller | \ logger.go:42: 16:17:39 | watcher-notification/1-deploy-with-notification | Already 2026-03-20 16:26:25.640071 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.640076 | controller | \ logger.go:42: 16:17:39 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.640080 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.640084 | controller | \ logger.go:42: 16:17:39 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.640088 | controller | head -1\n logger.go:42: 16:17:39 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.640093 | controller | | ++ grep -v '^$'\n logger.go:42: 16:17:39 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.640097 | controller | | + APIPOD=\n logger.go:42: 16:17:40 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.640101 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-20 16:26:25.640113 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-20 16:26:25.640118 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-20 16:26:25.640122 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.640126 | controller | |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-20 16:26:25.640131 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-20 16:26:25.640138 | controller | 16:17:40 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-20 16:26:25.640142 | controller | \ logger.go:42: 16:17:40 | watcher-notification/1-deploy-with-notification | Already 2026-03-20 16:26:25.640146 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.640151 | controller | \ logger.go:42: 16:17:40 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.640155 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.640159 | controller | \ logger.go:42: 16:17:40 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.640163 | controller | head -1\n logger.go:42: 16:17:40 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.640170 | controller | | ++ grep -v '^$'\n logger.go:42: 16:17:40 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.640174 | controller | | + APIPOD=\n logger.go:42: 16:17:41 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.640179 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-20 16:26:25.640183 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-20 16:26:25.640187 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-20 16:26:25.640191 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.640196 | controller | |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-20 16:26:25.640200 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-20 16:26:25.640204 | controller | 16:17:41 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-20 16:26:25.640209 | controller | \ logger.go:42: 16:17:41 | watcher-notification/1-deploy-with-notification | Already 2026-03-20 16:26:25.640213 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.640217 | controller | \ logger.go:42: 16:17:41 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.640221 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.640225 | controller | \ logger.go:42: 16:17:41 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.640230 | controller | head -1\n logger.go:42: 16:17:41 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.640234 | controller | | ++ grep -v '^$'\n logger.go:42: 16:17:42 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.640238 | controller | | + APIPOD=\n logger.go:42: 16:17:43 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.640242 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-20 16:26:25.640247 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-20 16:26:25.640251 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-20 16:26:25.640255 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.640259 | controller | |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-20 16:26:25.640264 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-20 16:26:25.640268 | controller | 16:17:43 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-20 16:26:25.640272 | controller | \ logger.go:42: 16:17:43 | watcher-notification/1-deploy-with-notification | Already 2026-03-20 16:26:25.640276 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.640281 | controller | \ logger.go:42: 16:17:43 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.640285 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.640289 | controller | \ logger.go:42: 16:17:43 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.640294 | controller | head -1\n logger.go:42: 16:17:43 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.640300 | controller | | ++ grep -v '^$'\n logger.go:42: 16:17:43 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.640336 | controller | | + APIPOD=\n logger.go:42: 16:17:44 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.640342 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-20 16:26:25.640348 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-20 16:26:25.640353 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-20 16:26:25.640357 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.640361 | controller | |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-20 16:26:25.640366 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-20 16:26:25.640370 | controller | 16:17:44 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-20 16:26:25.640375 | controller | \ logger.go:42: 16:17:44 | watcher-notification/1-deploy-with-notification | Already 2026-03-20 16:26:25.640380 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.640384 | controller | \ logger.go:42: 16:17:44 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.640388 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.640392 | controller | \ logger.go:42: 16:17:44 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.640396 | controller | head -1\n logger.go:42: 16:17:44 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.640401 | controller | | ++ grep -v '^$'\n logger.go:42: 16:17:44 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.640405 | controller | | + APIPOD=\n logger.go:42: 16:17:45 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.640409 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-20 16:26:25.640413 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-20 16:26:25.640418 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-20 16:26:25.640422 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.640426 | controller | |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-20 16:26:25.640431 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-20 16:26:25.640435 | controller | 16:17:45 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-20 16:26:25.640439 | controller | \ logger.go:42: 16:17:45 | watcher-notification/1-deploy-with-notification | Already 2026-03-20 16:26:25.640443 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.640448 | controller | \ logger.go:42: 16:17:45 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.640454 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.640458 | controller | \ logger.go:42: 16:17:45 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.640462 | controller | head -1\n logger.go:42: 16:17:45 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.640467 | controller | | ++ grep -v '^$'\n logger.go:42: 16:17:46 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.640471 | controller | | + APIPOD=\n logger.go:42: 16:17:47 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.640475 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-20 16:26:25.640479 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-20 16:26:25.640484 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-20 16:26:25.640488 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.640494 | controller | |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-20 16:26:25.640498 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-20 16:26:25.640502 | controller | 16:17:47 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-20 16:26:25.640506 | controller | \ logger.go:42: 16:17:47 | watcher-notification/1-deploy-with-notification | Already 2026-03-20 16:26:25.640510 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.640515 | controller | \ logger.go:42: 16:17:47 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.640519 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.640523 | controller | \ logger.go:42: 16:17:47 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.640535 | controller | head -1\n logger.go:42: 16:17:47 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.640540 | controller | | ++ grep -v '^$'\n logger.go:42: 16:17:47 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.640544 | controller | | + APIPOD=\n logger.go:42: 16:17:48 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.640548 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-20 16:26:25.640553 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-20 16:26:25.640557 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-20 16:26:25.640561 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.640565 | controller | |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-20 16:26:25.640570 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-20 16:26:25.640574 | controller | 16:17:48 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-20 16:26:25.640580 | controller | \ logger.go:42: 16:17:48 | watcher-notification/1-deploy-with-notification | Already 2026-03-20 16:26:25.640585 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.640589 | controller | \ logger.go:42: 16:17:48 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.640593 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.640597 | controller | \ logger.go:42: 16:17:48 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.640602 | controller | head -1\n logger.go:42: 16:17:48 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.640606 | controller | | ++ grep -v '^$'\n logger.go:42: 16:17:48 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.640610 | controller | | + APIPOD=\n logger.go:42: 16:17:49 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.640614 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-20 16:26:25.640619 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-20 16:26:25.640623 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-20 16:26:25.640627 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.640631 | controller | |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-20 16:26:25.640636 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-20 16:26:25.640640 | controller | 16:17:49 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-20 16:26:25.640646 | controller | \ logger.go:42: 16:17:49 | watcher-notification/1-deploy-with-notification | Already 2026-03-20 16:26:25.640651 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.640655 | controller | \ logger.go:42: 16:17:49 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.640660 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.640664 | controller | \ logger.go:42: 16:17:49 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.640668 | controller | head -1\n logger.go:42: 16:17:49 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.640673 | controller | | ++ grep -v '^$'\n logger.go:42: 16:17:50 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.640677 | controller | | + APIPOD=\n logger.go:42: 16:17:51 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.640681 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-20 16:26:25.640685 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-20 16:26:25.640689 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-20 16:26:25.640709 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.640714 | controller | |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-20 16:26:25.640718 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-20 16:26:25.640722 | controller | 16:17:51 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-20 16:26:25.640727 | controller | \ logger.go:42: 16:17:51 | watcher-notification/1-deploy-with-notification | Already 2026-03-20 16:26:25.640731 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.640743 | controller | \ logger.go:42: 16:17:51 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.640748 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.640753 | controller | \ logger.go:42: 16:17:51 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.640757 | controller | head -1\n logger.go:42: 16:17:51 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.640761 | controller | | ++ grep -v '^$'\n logger.go:42: 16:17:51 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.640767 | controller | | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 16:17:51 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.640772 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 16:17:51 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.640776 | controller | | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n 2026-03-20 16:26:25.640780 | controller | \ logger.go:42: 16:17:51 | watcher-notification/1-deploy-with-notification | +++ 2026-03-20 16:26:25.640785 | controller | oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-20 16:26:25.640789 | controller | \ logger.go:42: 16:17:51 | watcher-notification/1-deploy-with-notification | error: 2026-03-20 16:26:25.640793 | controller | unable to upgrade connection: container not found (\"watcher-api\")\n logger.go:42: 2026-03-20 16:26:25.640797 | controller | 16:17:51 | watcher-notification/1-deploy-with-notification | ++ echo\n logger.go:42: 2026-03-20 16:26:25.640802 | controller | 16:17:51 | watcher-notification/1-deploy-with-notification | + '[' 0 == 1 ']'\n 2026-03-20 16:26:25.640806 | controller | \ logger.go:42: 16:17:52 | watcher-notification/1-deploy-with-notification | running 2026-03-20 16:26:25.640812 | controller | command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc 2026-03-20 16:26:25.640817 | controller | get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep 2026-03-20 16:26:25.640821 | controller | -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh 2026-03-20 16:26:25.640825 | controller | -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) |grep 2026-03-20 16:26:25.640829 | controller | -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-20 16:26:25.640837 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-20 16:26:25.640841 | controller | 16:17:52 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-20 16:26:25.640845 | controller | \ logger.go:42: 16:17:52 | watcher-notification/1-deploy-with-notification | Already 2026-03-20 16:26:25.640850 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.640854 | controller | \ logger.go:42: 16:17:52 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.640858 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.640862 | controller | \ logger.go:42: 16:17:52 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.640867 | controller | head -1\n logger.go:42: 16:17:52 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.640872 | controller | | ++ grep -v '^$'\n logger.go:42: 16:17:53 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.640877 | controller | | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 16:17:53 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.640881 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 16:17:53 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.640885 | controller | | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n 2026-03-20 16:26:25.640889 | controller | \ logger.go:42: 16:17:53 | watcher-notification/1-deploy-with-notification | +++ 2026-03-20 16:26:25.640894 | controller | oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-20 16:26:25.640898 | controller | \ logger.go:42: 16:17:53 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.640903 | controller | echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.640907 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.640911 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.640916 | controller | = '[database]' connection = 'mysql+pymysql://watcher_a660:8502e801b18e815763860e7eefc70339@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-20 16:26:25.640920 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-20 16:26:25.640932 | controller | driver = messagingv2 transport_url = 'rabbit://default_user_I6f6zKOBLU76Pr74C5S:NMvNvliR9JfNJwTtRx2-upsPmWKjcAkC@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.640937 | controller | '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.640941 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.640945 | controller | project_domain_name = Default project_name = service user_domain_name = Default 2026-03-20 16:26:25.640950 | controller | password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 2026-03-20 16:26:25.640957 | controller | interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.640961 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-20 16:26:25.640965 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-20 16:26:25.640970 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-20 16:26:25.640974 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-20 16:26:25.640978 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-20 16:26:25.640982 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.640986 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.640991 | controller | tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc 2026-03-20 16:26:25.640995 | controller | port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' 2026-03-20 16:26:25.640999 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-20 16:26:25.641003 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.641008 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-20 16:26:25.641012 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-20 16:26:25.641016 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.641021 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-20 16:26:25.641025 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-20 16:26:25.641029 | controller | period = 900\n logger.go:42: 16:17:53 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.641033 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:17:54 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.641037 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-20 16:26:25.641042 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-20 16:26:25.641046 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-20 16:26:25.641050 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.641054 | controller | |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-20 16:26:25.641059 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-20 16:26:25.641063 | controller | 16:17:54 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-20 16:26:25.641067 | controller | \ logger.go:42: 16:17:54 | watcher-notification/1-deploy-with-notification | Already 2026-03-20 16:26:25.641071 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.641075 | controller | \ logger.go:42: 16:17:54 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.641080 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.641084 | controller | \ logger.go:42: 16:17:54 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.641088 | controller | head -1\n logger.go:42: 16:17:54 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.641094 | controller | | ++ grep -v '^$'\n logger.go:42: 16:17:54 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.641098 | controller | | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 16:17:54 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.641102 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 16:17:54 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.641114 | controller | | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n 2026-03-20 16:26:25.641118 | controller | \ logger.go:42: 16:17:54 | watcher-notification/1-deploy-with-notification | +++ 2026-03-20 16:26:25.641123 | controller | oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-20 16:26:25.641127 | controller | \ logger.go:42: 16:17:54 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.641131 | controller | echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.641136 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.641143 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.641148 | controller | = '[database]' connection = 'mysql+pymysql://watcher_a660:8502e801b18e815763860e7eefc70339@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-20 16:26:25.641152 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-20 16:26:25.641156 | controller | driver = messagingv2 transport_url = 'rabbit://default_user_I6f6zKOBLU76Pr74C5S:NMvNvliR9JfNJwTtRx2-upsPmWKjcAkC@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.641161 | controller | '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.641165 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.641169 | controller | project_domain_name = Default project_name = service user_domain_name = Default 2026-03-20 16:26:25.641173 | controller | password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 2026-03-20 16:26:25.641177 | controller | interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.641182 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-20 16:26:25.641186 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-20 16:26:25.641190 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-20 16:26:25.641194 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-20 16:26:25.641199 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-20 16:26:25.641203 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.641207 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.641211 | controller | tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc 2026-03-20 16:26:25.641215 | controller | port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' 2026-03-20 16:26:25.641220 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-20 16:26:25.641224 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.641228 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-20 16:26:25.641234 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-20 16:26:25.641239 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.641243 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-20 16:26:25.641247 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-20 16:26:25.641251 | controller | period = 900\n logger.go:42: 16:17:54 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.641256 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:17:55 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.641260 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-20 16:26:25.641264 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-20 16:26:25.641268 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-20 16:26:25.641273 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.641278 | controller | |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-20 16:26:25.641282 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-20 16:26:25.641294 | controller | 16:17:55 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-20 16:26:25.641299 | controller | \ logger.go:42: 16:17:56 | watcher-notification/1-deploy-with-notification | Already 2026-03-20 16:26:25.641303 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.641307 | controller | \ logger.go:42: 16:17:56 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.641312 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.641316 | controller | \ logger.go:42: 16:17:56 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.641320 | controller | head -1\n logger.go:42: 16:17:56 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.641325 | controller | | ++ grep -v '^$'\n logger.go:42: 16:17:56 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.641329 | controller | | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 16:17:56 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.641333 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 16:17:56 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.641339 | controller | | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n 2026-03-20 16:26:25.641344 | controller | \ logger.go:42: 16:17:56 | watcher-notification/1-deploy-with-notification | +++ 2026-03-20 16:26:25.641349 | controller | oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-20 16:26:25.641353 | controller | \ logger.go:42: 16:17:56 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.641358 | controller | echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.641364 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.641368 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.641373 | controller | = '[database]' connection = 'mysql+pymysql://watcher_a660:8502e801b18e815763860e7eefc70339@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-20 16:26:25.641379 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-20 16:26:25.641383 | controller | driver = messagingv2 transport_url = 'rabbit://default_user_I6f6zKOBLU76Pr74C5S:NMvNvliR9JfNJwTtRx2-upsPmWKjcAkC@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.641387 | controller | '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.641391 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.641396 | controller | project_domain_name = Default project_name = service user_domain_name = Default 2026-03-20 16:26:25.641400 | controller | password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 2026-03-20 16:26:25.641404 | controller | interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.641409 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-20 16:26:25.641413 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-20 16:26:25.641417 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-20 16:26:25.641421 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-20 16:26:25.641426 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-20 16:26:25.641430 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.641434 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.641438 | controller | tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc 2026-03-20 16:26:25.641444 | controller | port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' 2026-03-20 16:26:25.641448 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-20 16:26:25.641453 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.641457 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-20 16:26:25.641461 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-20 16:26:25.641473 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.641478 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-20 16:26:25.641482 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-20 16:26:25.641487 | controller | period = 900\n logger.go:42: 16:17:56 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.641491 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:17:57 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.641495 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-20 16:26:25.641499 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-20 16:26:25.641504 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-20 16:26:25.641508 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.641512 | controller | |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-20 16:26:25.641516 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-20 16:26:25.641523 | controller | 16:17:57 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-20 16:26:25.641527 | controller | \ logger.go:42: 16:17:57 | watcher-notification/1-deploy-with-notification | Already 2026-03-20 16:26:25.641531 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.641536 | controller | \ logger.go:42: 16:17:57 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.641540 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.641544 | controller | \ logger.go:42: 16:17:57 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.641548 | controller | head -1\n logger.go:42: 16:17:57 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.641553 | controller | | ++ grep -v '^$'\n logger.go:42: 16:17:57 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.641557 | controller | | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 16:17:57 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.641561 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 16:17:57 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.641565 | controller | | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n 2026-03-20 16:26:25.641570 | controller | \ logger.go:42: 16:17:57 | watcher-notification/1-deploy-with-notification | +++ 2026-03-20 16:26:25.641574 | controller | oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-20 16:26:25.641578 | controller | \ logger.go:42: 16:17:57 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.641582 | controller | echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.641587 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.641591 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.641595 | controller | = '[database]' connection = 'mysql+pymysql://watcher_a660:8502e801b18e815763860e7eefc70339@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-20 16:26:25.641599 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-20 16:26:25.641604 | controller | driver = messagingv2 transport_url = 'rabbit://default_user_I6f6zKOBLU76Pr74C5S:NMvNvliR9JfNJwTtRx2-upsPmWKjcAkC@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.641608 | controller | '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.641612 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.641616 | controller | project_domain_name = Default project_name = service user_domain_name = Default 2026-03-20 16:26:25.641621 | controller | password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 2026-03-20 16:26:25.641625 | controller | interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.641629 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-20 16:26:25.641633 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-20 16:26:25.641644 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-20 16:26:25.641648 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-20 16:26:25.641653 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-20 16:26:25.641659 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.641663 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.641668 | controller | tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc 2026-03-20 16:26:25.641672 | controller | port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' 2026-03-20 16:26:25.641676 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-20 16:26:25.641681 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.641685 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-20 16:26:25.641689 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-20 16:26:25.641709 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.641714 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-20 16:26:25.641719 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-20 16:26:25.641723 | controller | period = 900\n logger.go:42: 16:17:57 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.641727 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:17:59 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.641734 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-20 16:26:25.641739 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-20 16:26:25.641743 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-20 16:26:25.641747 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.641752 | controller | |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-20 16:26:25.641756 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-20 16:26:25.641760 | controller | 16:17:59 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-20 16:26:25.641765 | controller | \ logger.go:42: 16:17:59 | watcher-notification/1-deploy-with-notification | Already 2026-03-20 16:26:25.641769 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.641773 | controller | \ logger.go:42: 16:17:59 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.641777 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.641784 | controller | \ logger.go:42: 16:17:59 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.641788 | controller | head -1\n logger.go:42: 16:17:59 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.641792 | controller | | ++ grep -v '^$'\n logger.go:42: 16:17:59 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.641797 | controller | | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 16:17:59 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.641802 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 16:17:59 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.641806 | controller | | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n 2026-03-20 16:26:25.641810 | controller | \ logger.go:42: 16:17:59 | watcher-notification/1-deploy-with-notification | +++ 2026-03-20 16:26:25.641815 | controller | oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-20 16:26:25.641821 | controller | \ logger.go:42: 16:17:59 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.641825 | controller | echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.641829 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.641834 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.641838 | controller | = '[database]' connection = 'mysql+pymysql://watcher_a660:8502e801b18e815763860e7eefc70339@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-20 16:26:25.641849 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-20 16:26:25.641854 | controller | driver = messagingv2 transport_url = 'rabbit://default_user_I6f6zKOBLU76Pr74C5S:NMvNvliR9JfNJwTtRx2-upsPmWKjcAkC@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.641858 | controller | '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.641862 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.641867 | controller | project_domain_name = Default project_name = service user_domain_name = Default 2026-03-20 16:26:25.641871 | controller | password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 2026-03-20 16:26:25.641879 | controller | interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.641884 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-20 16:26:25.641888 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-20 16:26:25.641892 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-20 16:26:25.641897 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-20 16:26:25.641901 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-20 16:26:25.641905 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.641909 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.641914 | controller | tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc 2026-03-20 16:26:25.641918 | controller | port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' 2026-03-20 16:26:25.641922 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-20 16:26:25.641926 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.641931 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-20 16:26:25.641935 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-20 16:26:25.641939 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.641943 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-20 16:26:25.641948 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-20 16:26:25.641952 | controller | period = 900\n logger.go:42: 16:17:59 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.641958 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:18:00 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.641962 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-20 16:26:25.641966 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-20 16:26:25.641971 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-20 16:26:25.641975 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.641979 | controller | |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-20 16:26:25.641983 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-20 16:26:25.641987 | controller | 16:18:00 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-20 16:26:25.641992 | controller | \ logger.go:42: 16:18:00 | watcher-notification/1-deploy-with-notification | Already 2026-03-20 16:26:25.641996 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.642000 | controller | \ logger.go:42: 16:18:00 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.642004 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.642009 | controller | \ logger.go:42: 16:18:00 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.642013 | controller | head -1\n logger.go:42: 16:18:00 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.642017 | controller | | ++ grep -v '^$'\n logger.go:42: 16:18:00 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.642021 | controller | | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 16:18:00 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.642033 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 16:18:00 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.642038 | controller | | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n 2026-03-20 16:26:25.642043 | controller | \ logger.go:42: 16:18:00 | watcher-notification/1-deploy-with-notification | +++ 2026-03-20 16:26:25.642047 | controller | oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-20 16:26:25.642052 | controller | \ logger.go:42: 16:18:01 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.642056 | controller | echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.642060 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.642064 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.642069 | controller | = '[database]' connection = 'mysql+pymysql://watcher_a660:8502e801b18e815763860e7eefc70339@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-20 16:26:25.642073 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-20 16:26:25.642077 | controller | driver = messagingv2 transport_url = 'rabbit://default_user_I6f6zKOBLU76Pr74C5S:NMvNvliR9JfNJwTtRx2-upsPmWKjcAkC@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.642084 | controller | '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.642088 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.642093 | controller | project_domain_name = Default project_name = service user_domain_name = Default 2026-03-20 16:26:25.642099 | controller | password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 2026-03-20 16:26:25.642103 | controller | interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.642108 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-20 16:26:25.642112 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-20 16:26:25.642116 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-20 16:26:25.642120 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-20 16:26:25.642125 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-20 16:26:25.642129 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.642133 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.642137 | controller | tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc 2026-03-20 16:26:25.642142 | controller | port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' 2026-03-20 16:26:25.642146 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-20 16:26:25.642150 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.642154 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-20 16:26:25.642159 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-20 16:26:25.642163 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.642167 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-20 16:26:25.642172 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-20 16:26:25.642176 | controller | period = 900\n logger.go:42: 16:18:01 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.642180 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:18:02 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.642184 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-20 16:26:25.642189 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-20 16:26:25.642193 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-20 16:26:25.642197 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.642208 | controller | |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-20 16:26:25.642212 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-20 16:26:25.642217 | controller | 16:18:02 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-20 16:26:25.642221 | controller | \ logger.go:42: 16:18:02 | watcher-notification/1-deploy-with-notification | Already 2026-03-20 16:26:25.642225 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.642230 | controller | \ logger.go:42: 16:18:02 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.642234 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.642238 | controller | \ logger.go:42: 16:18:02 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.642244 | controller | head -1\n logger.go:42: 16:18:02 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.642248 | controller | | ++ grep -v '^$'\n logger.go:42: 16:18:02 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.642253 | controller | | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 16:18:02 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.642257 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 16:18:02 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.642261 | controller | | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n 2026-03-20 16:26:25.642265 | controller | \ logger.go:42: 16:18:02 | watcher-notification/1-deploy-with-notification | +++ 2026-03-20 16:26:25.642272 | controller | oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-20 16:26:25.642276 | controller | \ logger.go:42: 16:18:02 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.642293 | controller | echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.642298 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.642302 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.642306 | controller | = '[database]' connection = 'mysql+pymysql://watcher_a660:8502e801b18e815763860e7eefc70339@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-20 16:26:25.642310 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-20 16:26:25.642315 | controller | driver = messagingv2 transport_url = 'rabbit://default_user_I6f6zKOBLU76Pr74C5S:NMvNvliR9JfNJwTtRx2-upsPmWKjcAkC@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.642319 | controller | '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.642323 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.642327 | controller | project_domain_name = Default project_name = service user_domain_name = Default 2026-03-20 16:26:25.642332 | controller | password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 2026-03-20 16:26:25.642338 | controller | interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.642342 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-20 16:26:25.642347 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-20 16:26:25.642351 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-20 16:26:25.642355 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-20 16:26:25.642359 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-20 16:26:25.642363 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.642368 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.642372 | controller | tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc 2026-03-20 16:26:25.642376 | controller | port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' 2026-03-20 16:26:25.642382 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-20 16:26:25.642539 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.642547 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-20 16:26:25.642558 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-20 16:26:25.642563 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.642567 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-20 16:26:25.642571 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-20 16:26:25.642575 | controller | period = 900\n logger.go:42: 16:18:02 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.642580 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:18:03 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.642584 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-20 16:26:25.642588 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-20 16:26:25.642592 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-20 16:26:25.642596 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.642601 | controller | |grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+=') 2026-03-20 16:26:25.642605 | controller | == 1 ]\n else\n exit 1\n fi\n ]\n logger.go:42: 2026-03-20 16:26:25.642609 | controller | 16:18:03 | watcher-notification/1-deploy-with-notification | + oc project watcher-kuttl-default\n 2026-03-20 16:26:25.642613 | controller | \ logger.go:42: 16:18:03 | watcher-notification/1-deploy-with-notification | Already 2026-03-20 16:26:25.642617 | controller | on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.642621 | controller | \ logger.go:42: 16:18:03 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.642626 | controller | oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.642630 | controller | \ logger.go:42: 16:18:03 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.642634 | controller | head -1\n logger.go:42: 16:18:03 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.642638 | controller | | ++ grep -v '^$'\n logger.go:42: 16:18:04 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.642642 | controller | | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 16:18:04 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.642646 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 16:18:04 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.642651 | controller | | ++ grep -czPo '\\[oslo_messaging_notifications\\]\\s+driver\\s+=\\s+messagingv2\\s+transport_url\\s+='\n 2026-03-20 16:26:25.642655 | controller | \ logger.go:42: 16:18:04 | watcher-notification/1-deploy-with-notification | +++ 2026-03-20 16:26:25.642659 | controller | oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-20 16:26:25.642663 | controller | \ logger.go:42: 16:18:04 | watcher-notification/1-deploy-with-notification | ++ 2026-03-20 16:26:25.642668 | controller | echo '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.642672 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.642685 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.642689 | controller | = '[database]' connection = 'mysql+pymysql://watcher_a660:8502e801b18e815763860e7eefc70339@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-20 16:26:25.642709 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-20 16:26:25.642714 | controller | driver = messagingv2 transport_url = 'rabbit://default_user_I6f6zKOBLU76Pr74C5S:NMvNvliR9JfNJwTtRx2-upsPmWKjcAkC@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.642718 | controller | '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.642723 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.642727 | controller | project_domain_name = Default project_name = service user_domain_name = Default 2026-03-20 16:26:25.642731 | controller | password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 2026-03-20 16:26:25.642736 | controller | interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.642740 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-20 16:26:25.642751 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-20 16:26:25.642756 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-20 16:26:25.642760 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-20 16:26:25.642764 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-20 16:26:25.642769 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.642773 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.642777 | controller | tls_enabled=true memcache_dead_retry = 30 '[prometheus_client]' host = metric-storage-prometheus.watcher-kuttl-default.svc 2026-03-20 16:26:25.642781 | controller | port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' 2026-03-20 16:26:25.642785 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-20 16:26:25.642790 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.642794 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-20 16:26:25.642798 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-20 16:26:25.642802 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.642806 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-20 16:26:25.642811 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-20 16:26:25.642815 | controller | period = 900\n logger.go:42: 16:18:04 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.642819 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:18:04 | watcher-notification/1-deploy-with-notification 2026-03-20 16:26:25.642823 | controller | | test step completed 1-deploy-with-notification\n logger.go:42: 16:18:04 | watcher-notification/2-cleanup-watcher 2026-03-20 16:26:25.642827 | controller | | starting test step 2-cleanup-watcher\n logger.go:42: 16:18:11 | watcher-notification/2-cleanup-watcher 2026-03-20 16:26:25.642832 | controller | | test step completed 2-cleanup-watcher\n logger.go:42: 16:18:11 | watcher-notification 2026-03-20 16:26:25.642836 | controller | | skipping kubernetes event logging\n=== CONT kuttl/harness/watcher-topology\n 2026-03-20 16:26:25.642848 | controller | \ logger.go:42: 16:18:11 | watcher-topology | Skipping creation of user-supplied 2026-03-20 16:26:25.642855 | controller | namespace: watcher-kuttl-default\n logger.go:42: 16:18:11 | watcher-topology/0-cleanup-watcher 2026-03-20 16:26:25.642859 | controller | | starting test step 0-cleanup-watcher\n logger.go:42: 16:18:11 | watcher-topology/0-cleanup-watcher 2026-03-20 16:26:25.642863 | controller | | test step completed 0-cleanup-watcher\n logger.go:42: 16:18:11 | watcher-topology/1-deploy-with-topology 2026-03-20 16:26:25.642868 | controller | | starting test step 1-deploy-with-topology\n logger.go:42: 16:18:11 | watcher-topology/1-deploy-with-topology 2026-03-20 16:26:25.642872 | controller | | Topology:watcher-kuttl-default/watcher-api created\n logger.go:42: 16:18:11 2026-03-20 16:26:25.642876 | controller | | watcher-topology/1-deploy-with-topology | Watcher:watcher-kuttl-default/watcher-kuttl 2026-03-20 16:26:25.642880 | controller | created\n logger.go:42: 16:18:36 | watcher-topology/1-deploy-with-topology | 2026-03-20 16:26:25.642885 | controller | test step completed 1-deploy-with-topology\n logger.go:42: 16:18:36 | watcher-topology/2-cleanup-watcher 2026-03-20 16:26:25.642889 | controller | | starting test step 2-cleanup-watcher\n logger.go:42: 16:18:36 | watcher-topology/2-cleanup-watcher 2026-03-20 16:26:25.642893 | controller | | running command: [sh -c set -ex\n oc exec -n watcher-kuttl-default openstackclient 2026-03-20 16:26:25.642899 | controller | -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 2026-03-20 16:26:25.642904 | controller | ]\n ]\n logger.go:42: 16:18:36 | watcher-topology/2-cleanup-watcher | 2026-03-20 16:26:25.642908 | controller | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-20 16:26:25.642912 | controller | value -c Name -c Type\n logger.go:42: 16:18:36 | watcher-topology/2-cleanup-watcher 2026-03-20 16:26:25.642916 | controller | | ++ grep -c '^watcher'\n logger.go:42: 16:18:39 | watcher-topology/2-cleanup-watcher 2026-03-20 16:26:25.642920 | controller | | + '[' 1 == 0 ']'\n logger.go:42: 16:18:40 | watcher-topology/2-cleanup-watcher 2026-03-20 16:26:25.642925 | controller | | running command: [sh -c set -ex\n oc exec -n watcher-kuttl-default openstackclient 2026-03-20 16:26:25.642929 | controller | -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 2026-03-20 16:26:25.642933 | controller | ]\n ]\n logger.go:42: 16:18:40 | watcher-topology/2-cleanup-watcher | 2026-03-20 16:26:25.642944 | controller | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-20 16:26:25.642949 | controller | value -c Name -c Type\n logger.go:42: 16:18:40 | watcher-topology/2-cleanup-watcher 2026-03-20 16:26:25.642953 | controller | | ++ grep -c '^watcher'\n logger.go:42: 16:18:42 | watcher-topology/2-cleanup-watcher 2026-03-20 16:26:25.642957 | controller | | + '[' 0 == 0 ']'\n logger.go:42: 16:18:42 | watcher-topology/2-cleanup-watcher 2026-03-20 16:26:25.642962 | controller | | test step completed 2-cleanup-watcher\n logger.go:42: 16:18:42 | watcher-topology 2026-03-20 16:26:25.642966 | controller | | skipping kubernetes event logging\n=== CONT kuttl/harness/watcher-tls-certs-change\n 2026-03-20 16:26:25.642970 | controller | \ logger.go:42: 16:18:42 | watcher-tls-certs-change | Skipping creation of user-supplied 2026-03-20 16:26:25.642974 | controller | namespace: watcher-kuttl-default\n logger.go:42: 16:18:42 | watcher-tls-certs-change/0-cleanup-watcher 2026-03-20 16:26:25.642978 | controller | | starting test step 0-cleanup-watcher\n logger.go:42: 16:18:42 | watcher-tls-certs-change/0-cleanup-watcher 2026-03-20 16:26:25.642982 | controller | | test step completed 0-cleanup-watcher\n logger.go:42: 16:18:42 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.642987 | controller | | starting test step 1-deploy-with-tlse\n logger.go:42: 16:18:42 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.642991 | controller | | Certificate:watcher-kuttl-default/watcher-internal-svc created\n logger.go:42: 2026-03-20 16:26:25.642995 | controller | 16:18:42 | watcher-tls-certs-change/1-deploy-with-tlse | Certificate:watcher-kuttl-default/watcher-public-svc 2026-03-20 16:26:25.643002 | controller | created\n logger.go:42: 16:18:42 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.643006 | controller | | Watcher:watcher-kuttl-default/watcher-kuttl created\n logger.go:42: 16:18:43 2026-03-20 16:26:25.643010 | controller | | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c set -euxo 2026-03-20 16:26:25.643014 | controller | pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-20 16:26:25.643019 | controller | cat /etc/pki/tls/certs/public.crt)\n public_secret_cert=$(oc get -n $NAMESPACE 2026-03-20 16:26:25.643023 | controller | secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n 2026-03-20 16:26:25.643027 | controller | \ # ensure that the svc secret and cert secret match\n if [ \"${public_svc_cert}\" 2026-03-20 16:26:25.643031 | controller | != \"${public_secret_cert}\" ]; then\n exit 1\n fi\n \n internal_svc_cert=$(oc 2026-03-20 16:26:25.643035 | controller | rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n 2026-03-20 16:26:25.643040 | controller | \ internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-20 16:26:25.643044 | controller | -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc 2026-03-20 16:26:25.643048 | controller | secret and cert secret match\n if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" 2026-03-20 16:26:25.643052 | controller | ]; then\n exit 1\n fi\n ]\n logger.go:42: 16:18:43 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.643056 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-20 16:26:25.643061 | controller | \ logger.go:42: 16:18:43 | watcher-tls-certs-change/1-deploy-with-tlse | Error 2026-03-20 16:26:25.643065 | controller | from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n logger.go:42: 2026-03-20 16:26:25.643079 | controller | 16:18:43 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n logger.go:42: 2026-03-20 16:26:25.643084 | controller | 16:18:44 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c 2026-03-20 16:26:25.643088 | controller | set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-20 16:26:25.643093 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n public_secret_cert=$(oc 2026-03-20 16:26:25.643097 | controller | get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-20 16:26:25.643101 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-20 16:26:25.643105 | controller | \ if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 2026-03-20 16:26:25.643109 | controller | 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-20 16:26:25.643114 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n internal_secret_cert=$(oc 2026-03-20 16:26:25.643119 | controller | get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-20 16:26:25.643124 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-20 16:26:25.643128 | controller | \ if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 2026-03-20 16:26:25.643132 | controller | 1\n fi\n ]\n logger.go:42: 16:18:44 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.643144 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-20 16:26:25.643148 | controller | \ logger.go:42: 16:18:44 | watcher-tls-certs-change/1-deploy-with-tlse | Error 2026-03-20 16:26:25.643153 | controller | from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n logger.go:42: 2026-03-20 16:26:25.643157 | controller | 16:18:44 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n logger.go:42: 2026-03-20 16:26:25.643164 | controller | 16:18:45 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c 2026-03-20 16:26:25.643169 | controller | set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-20 16:26:25.643173 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n public_secret_cert=$(oc 2026-03-20 16:26:25.643177 | controller | get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-20 16:26:25.643181 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-20 16:26:25.643185 | controller | \ if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 2026-03-20 16:26:25.643190 | controller | 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-20 16:26:25.643194 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n internal_secret_cert=$(oc 2026-03-20 16:26:25.643198 | controller | get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-20 16:26:25.643202 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-20 16:26:25.643206 | controller | \ if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 2026-03-20 16:26:25.643211 | controller | 1\n fi\n ]\n logger.go:42: 16:18:45 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.643215 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-20 16:26:25.643219 | controller | \ logger.go:42: 16:18:45 | watcher-tls-certs-change/1-deploy-with-tlse | Error 2026-03-20 16:26:25.643223 | controller | from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n logger.go:42: 2026-03-20 16:26:25.643227 | controller | 16:18:45 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n logger.go:42: 2026-03-20 16:26:25.643232 | controller | 16:18:46 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c 2026-03-20 16:26:25.643236 | controller | set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-20 16:26:25.643240 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n public_secret_cert=$(oc 2026-03-20 16:26:25.643244 | controller | get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-20 16:26:25.643248 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-20 16:26:25.643252 | controller | \ if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 2026-03-20 16:26:25.643257 | controller | 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-20 16:26:25.643261 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n internal_secret_cert=$(oc 2026-03-20 16:26:25.643265 | controller | get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-20 16:26:25.643269 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-20 16:26:25.643273 | controller | \ if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 2026-03-20 16:26:25.643278 | controller | 1\n fi\n ]\n logger.go:42: 16:18:46 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.643282 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-20 16:26:25.643286 | controller | \ logger.go:42: 16:18:46 | watcher-tls-certs-change/1-deploy-with-tlse | Error 2026-03-20 16:26:25.643290 | controller | from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n logger.go:42: 2026-03-20 16:26:25.643294 | controller | 16:18:46 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n logger.go:42: 2026-03-20 16:26:25.643299 | controller | 16:18:48 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c 2026-03-20 16:26:25.643306 | controller | set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-20 16:26:25.643310 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n public_secret_cert=$(oc 2026-03-20 16:26:25.643314 | controller | get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-20 16:26:25.643318 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-20 16:26:25.643325 | controller | \ if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 2026-03-20 16:26:25.643329 | controller | 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-20 16:26:25.643333 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n internal_secret_cert=$(oc 2026-03-20 16:26:25.643345 | controller | get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-20 16:26:25.643349 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-20 16:26:25.643353 | controller | \ if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 2026-03-20 16:26:25.643358 | controller | 1\n fi\n ]\n logger.go:42: 16:18:48 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.643362 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-20 16:26:25.643366 | controller | \ logger.go:42: 16:18:48 | watcher-tls-certs-change/1-deploy-with-tlse | Error 2026-03-20 16:26:25.643370 | controller | from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n logger.go:42: 2026-03-20 16:26:25.643374 | controller | 16:18:48 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n logger.go:42: 2026-03-20 16:26:25.643379 | controller | 16:18:49 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c 2026-03-20 16:26:25.643383 | controller | set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-20 16:26:25.643387 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n public_secret_cert=$(oc 2026-03-20 16:26:25.643391 | controller | get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-20 16:26:25.643395 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-20 16:26:25.643400 | controller | \ if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 2026-03-20 16:26:25.643404 | controller | 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-20 16:26:25.643408 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n internal_secret_cert=$(oc 2026-03-20 16:26:25.643412 | controller | get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-20 16:26:25.643416 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-20 16:26:25.643420 | controller | \ if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 2026-03-20 16:26:25.643425 | controller | 1\n fi\n ]\n logger.go:42: 16:18:49 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.643429 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-20 16:26:25.643433 | controller | \ logger.go:42: 16:18:50 | watcher-tls-certs-change/1-deploy-with-tlse | Error 2026-03-20 16:26:25.643439 | controller | from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n logger.go:42: 2026-03-20 16:26:25.643444 | controller | 16:18:50 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n logger.go:42: 2026-03-20 16:26:25.643448 | controller | 16:18:51 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c 2026-03-20 16:26:25.643452 | controller | set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-20 16:26:25.643458 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n public_secret_cert=$(oc 2026-03-20 16:26:25.643463 | controller | get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-20 16:26:25.643467 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-20 16:26:25.643471 | controller | \ if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 2026-03-20 16:26:25.643475 | controller | 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-20 16:26:25.643479 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n internal_secret_cert=$(oc 2026-03-20 16:26:25.643483 | controller | get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-20 16:26:25.643488 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-20 16:26:25.643492 | controller | \ if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 2026-03-20 16:26:25.643496 | controller | 1\n fi\n ]\n logger.go:42: 16:18:51 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.643500 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-20 16:26:25.643504 | controller | \ logger.go:42: 16:18:51 | watcher-tls-certs-change/1-deploy-with-tlse | Error 2026-03-20 16:26:25.643509 | controller | from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n logger.go:42: 2026-03-20 16:26:25.643513 | controller | 16:18:51 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n logger.go:42: 2026-03-20 16:26:25.643517 | controller | 16:18:52 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c 2026-03-20 16:26:25.643521 | controller | set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-20 16:26:25.643527 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n public_secret_cert=$(oc 2026-03-20 16:26:25.643560 | controller | get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-20 16:26:25.643566 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-20 16:26:25.643571 | controller | \ if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 2026-03-20 16:26:25.643576 | controller | 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-20 16:26:25.643581 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n internal_secret_cert=$(oc 2026-03-20 16:26:25.643586 | controller | get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-20 16:26:25.643591 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-20 16:26:25.643596 | controller | \ if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 2026-03-20 16:26:25.643600 | controller | 1\n fi\n ]\n logger.go:42: 16:18:52 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.643605 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-20 16:26:25.643610 | controller | \ logger.go:42: 16:18:52 | watcher-tls-certs-change/1-deploy-with-tlse | Error 2026-03-20 16:26:25.643615 | controller | from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n logger.go:42: 2026-03-20 16:26:25.643622 | controller | 16:18:52 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n logger.go:42: 2026-03-20 16:26:25.643627 | controller | 16:18:53 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c 2026-03-20 16:26:25.643632 | controller | set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-20 16:26:25.643637 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n public_secret_cert=$(oc 2026-03-20 16:26:25.643642 | controller | get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-20 16:26:25.643649 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-20 16:26:25.643653 | controller | \ if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 2026-03-20 16:26:25.643658 | controller | 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-20 16:26:25.643663 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n internal_secret_cert=$(oc 2026-03-20 16:26:25.643668 | controller | get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-20 16:26:25.643673 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-20 16:26:25.643678 | controller | \ if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 2026-03-20 16:26:25.643683 | controller | 1\n fi\n ]\n logger.go:42: 16:18:53 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.643688 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-20 16:26:25.643704 | controller | \ logger.go:42: 16:18:53 | watcher-tls-certs-change/1-deploy-with-tlse | Error 2026-03-20 16:26:25.643712 | controller | from server (NotFound): pods \"watcher-kuttl-api-0\" not found\n logger.go:42: 2026-03-20 16:26:25.643717 | controller | 16:18:53 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n logger.go:42: 2026-03-20 16:26:25.643722 | controller | 16:18:55 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c 2026-03-20 16:26:25.643727 | controller | set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-20 16:26:25.643732 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n public_secret_cert=$(oc 2026-03-20 16:26:25.643736 | controller | get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-20 16:26:25.643741 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-20 16:26:25.643746 | controller | \ if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 2026-03-20 16:26:25.643751 | controller | 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-20 16:26:25.643756 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n internal_secret_cert=$(oc 2026-03-20 16:26:25.643761 | controller | get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-20 16:26:25.643766 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-20 16:26:25.643770 | controller | \ if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 2026-03-20 16:26:25.643775 | controller | 1\n fi\n ]\n logger.go:42: 16:18:55 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.643780 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-20 16:26:25.643785 | controller | \ logger.go:42: 16:18:55 | watcher-tls-certs-change/1-deploy-with-tlse | error: 2026-03-20 16:26:25.643790 | controller | unable to upgrade connection: container not found (\"watcher-api\")\n logger.go:42: 2026-03-20 16:26:25.643803 | controller | 16:18:55 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=\n logger.go:42: 2026-03-20 16:26:25.643808 | controller | 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | running command: [sh -c 2026-03-20 16:26:25.643815 | controller | set -euxo pipefail\n public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-20 16:26:25.643819 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n public_secret_cert=$(oc 2026-03-20 16:26:25.643823 | controller | get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-20 16:26:25.643827 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-20 16:26:25.643834 | controller | \ if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" ]; then\n exit 2026-03-20 16:26:25.643838 | controller | 1\n fi\n \n internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api 2026-03-20 16:26:25.643842 | controller | watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n internal_secret_cert=$(oc 2026-03-20 16:26:25.643846 | controller | get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-20 16:26:25.643850 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-20 16:26:25.643854 | controller | \ if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" ]; then\n exit 2026-03-20 16:26:25.643859 | controller | 1\n fi\n ]\n logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.643863 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-20 16:26:25.643867 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='-----BEGIN 2026-03-20 16:26:25.643871 | controller | CERTIFICATE-----\n logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.643875 | controller | | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY\n logger.go:42: 2026-03-20 16:26:25.643880 | controller | 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx\n 2026-03-20 16:26:25.643884 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu\n 2026-03-20 16:26:25.643888 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH\n 2026-03-20 16:26:25.643892 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol\n 2026-03-20 16:26:25.643897 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X\n 2026-03-20 16:26:25.643901 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI\n 2026-03-20 16:26:25.643905 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28\n 2026-03-20 16:26:25.643909 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww\n 2026-03-20 16:26:25.643915 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/\n 2026-03-20 16:26:25.643919 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy\n 2026-03-20 16:26:25.643923 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH\n 2026-03-20 16:26:25.643928 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp\n 2026-03-20 16:26:25.643932 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6\n 2026-03-20 16:26:25.643936 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv\n 2026-03-20 16:26:25.643940 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl\n 2026-03-20 16:26:25.643944 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs\n 2026-03-20 16:26:25.643958 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0\n 2026-03-20 16:26:25.643963 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9\n 2026-03-20 16:26:25.643967 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=\n 2026-03-20 16:26:25.643972 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.643976 | controller | CERTIFICATE-----'\n logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.643980 | controller | | ++ base64 --decode\n logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.643984 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n 2026-03-20 16:26:25.643989 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert='-----BEGIN 2026-03-20 16:26:25.643993 | controller | CERTIFICATE-----\n logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.643997 | controller | | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY\n logger.go:42: 2026-03-20 16:26:25.644001 | controller | 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx\n 2026-03-20 16:26:25.644005 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu\n 2026-03-20 16:26:25.644010 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH\n 2026-03-20 16:26:25.644014 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol\n 2026-03-20 16:26:25.644018 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X\n 2026-03-20 16:26:25.644022 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI\n 2026-03-20 16:26:25.644029 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28\n 2026-03-20 16:26:25.644033 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww\n 2026-03-20 16:26:25.644037 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/\n 2026-03-20 16:26:25.644041 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy\n 2026-03-20 16:26:25.644046 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH\n 2026-03-20 16:26:25.644050 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp\n 2026-03-20 16:26:25.644054 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6\n 2026-03-20 16:26:25.644058 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv\n 2026-03-20 16:26:25.644062 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl\n 2026-03-20 16:26:25.644068 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs\n 2026-03-20 16:26:25.644073 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0\n 2026-03-20 16:26:25.644077 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9\n 2026-03-20 16:26:25.644081 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=\n 2026-03-20 16:26:25.644085 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.644090 | controller | CERTIFICATE-----'\n logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.644094 | controller | | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.644106 | controller | | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY\n logger.go:42: 2026-03-20 16:26:25.644110 | controller | 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx\n 2026-03-20 16:26:25.644115 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu\n 2026-03-20 16:26:25.644119 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH\n 2026-03-20 16:26:25.644123 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol\n 2026-03-20 16:26:25.644127 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X\n 2026-03-20 16:26:25.644131 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI\n 2026-03-20 16:26:25.644136 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28\n 2026-03-20 16:26:25.644140 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww\n 2026-03-20 16:26:25.644144 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/\n 2026-03-20 16:26:25.644148 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy\n 2026-03-20 16:26:25.644155 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH\n 2026-03-20 16:26:25.644159 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp\n 2026-03-20 16:26:25.644163 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6\n 2026-03-20 16:26:25.644167 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv\n 2026-03-20 16:26:25.644172 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl\n 2026-03-20 16:26:25.644176 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs\n 2026-03-20 16:26:25.644183 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0\n 2026-03-20 16:26:25.644187 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9\n 2026-03-20 16:26:25.644192 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=\n 2026-03-20 16:26:25.644196 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.644200 | controller | CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 16:18:56 2026-03-20 16:26:25.644204 | controller | | watcher-tls-certs-change/1-deploy-with-tlse | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY\n 2026-03-20 16:26:25.644208 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx\n 2026-03-20 16:26:25.644213 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu\n 2026-03-20 16:26:25.644217 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH\n 2026-03-20 16:26:25.644221 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol\n 2026-03-20 16:26:25.644225 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X\n 2026-03-20 16:26:25.644229 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI\n 2026-03-20 16:26:25.644241 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28\n 2026-03-20 16:26:25.644245 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww\n 2026-03-20 16:26:25.644250 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/\n 2026-03-20 16:26:25.644254 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy\n 2026-03-20 16:26:25.644258 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH\n 2026-03-20 16:26:25.644262 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp\n 2026-03-20 16:26:25.644266 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6\n 2026-03-20 16:26:25.644273 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv\n 2026-03-20 16:26:25.644277 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl\n 2026-03-20 16:26:25.644282 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs\n 2026-03-20 16:26:25.644286 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0\n 2026-03-20 16:26:25.644290 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9\n 2026-03-20 16:26:25.644297 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=\n 2026-03-20 16:26:25.644301 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.644306 | controller | CERTIFICATE-----' ']'\n logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.644310 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n 2026-03-20 16:26:25.644314 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert='-----BEGIN 2026-03-20 16:26:25.644318 | controller | CERTIFICATE-----\n logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.644323 | controller | | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-20 16:26:25.644327 | controller | 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx\n 2026-03-20 16:26:25.644331 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb\n 2026-03-20 16:26:25.644335 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2\n 2026-03-20 16:26:25.644339 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro\n 2026-03-20 16:26:25.644344 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671\n 2026-03-20 16:26:25.644348 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B\n 2026-03-20 16:26:25.644352 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ\n 2026-03-20 16:26:25.644356 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-20 16:26:25.644360 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E\n 2026-03-20 16:26:25.644364 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-20 16:26:25.644368 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J\n 2026-03-20 16:26:25.644380 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU\n 2026-03-20 16:26:25.644385 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq\n 2026-03-20 16:26:25.644389 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6\n 2026-03-20 16:26:25.644393 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3\n 2026-03-20 16:26:25.644398 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk\n 2026-03-20 16:26:25.644402 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb\n 2026-03-20 16:26:25.644408 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE\n 2026-03-20 16:26:25.644414 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==\n 2026-03-20 16:26:25.644418 | controller | \ logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.644422 | controller | CERTIFICATE-----'\n logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.644427 | controller | | ++ base64 --decode\n logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.644431 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n 2026-03-20 16:26:25.644435 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert='-----BEGIN 2026-03-20 16:26:25.644439 | controller | CERTIFICATE-----\n logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.644443 | controller | | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-20 16:26:25.644448 | controller | 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx\n 2026-03-20 16:26:25.644452 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb\n 2026-03-20 16:26:25.644456 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2\n 2026-03-20 16:26:25.644460 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro\n 2026-03-20 16:26:25.644464 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671\n 2026-03-20 16:26:25.644469 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B\n 2026-03-20 16:26:25.644473 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ\n 2026-03-20 16:26:25.644477 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-20 16:26:25.644481 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E\n 2026-03-20 16:26:25.644485 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-20 16:26:25.644490 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J\n 2026-03-20 16:26:25.644494 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU\n 2026-03-20 16:26:25.644498 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq\n 2026-03-20 16:26:25.644502 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6\n 2026-03-20 16:26:25.644506 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3\n 2026-03-20 16:26:25.644518 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk\n 2026-03-20 16:26:25.644522 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb\n 2026-03-20 16:26:25.644528 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE\n 2026-03-20 16:26:25.644533 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==\n 2026-03-20 16:26:25.644537 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.644541 | controller | CERTIFICATE-----'\n logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.644545 | controller | | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.644550 | controller | | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-20 16:26:25.644554 | controller | 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx\n 2026-03-20 16:26:25.644558 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb\n 2026-03-20 16:26:25.644564 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2\n 2026-03-20 16:26:25.644568 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro\n 2026-03-20 16:26:25.644573 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671\n 2026-03-20 16:26:25.644577 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B\n 2026-03-20 16:26:25.644581 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ\n 2026-03-20 16:26:25.644585 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-20 16:26:25.644589 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E\n 2026-03-20 16:26:25.644594 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-20 16:26:25.644598 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J\n 2026-03-20 16:26:25.644602 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU\n 2026-03-20 16:26:25.644606 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq\n 2026-03-20 16:26:25.644610 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6\n 2026-03-20 16:26:25.644615 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3\n 2026-03-20 16:26:25.644619 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk\n 2026-03-20 16:26:25.644623 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb\n 2026-03-20 16:26:25.644627 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE\n 2026-03-20 16:26:25.644631 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==\n 2026-03-20 16:26:25.644638 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.644642 | controller | CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 16:18:57 2026-03-20 16:26:25.644646 | controller | | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw\n 2026-03-20 16:26:25.644657 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx\n 2026-03-20 16:26:25.644661 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb\n 2026-03-20 16:26:25.644666 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2\n 2026-03-20 16:26:25.644670 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro\n 2026-03-20 16:26:25.644676 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671\n 2026-03-20 16:26:25.644680 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B\n 2026-03-20 16:26:25.644685 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ\n 2026-03-20 16:26:25.644689 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-20 16:26:25.644706 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E\n 2026-03-20 16:26:25.644712 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-20 16:26:25.644716 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J\n 2026-03-20 16:26:25.644721 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU\n 2026-03-20 16:26:25.644725 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq\n 2026-03-20 16:26:25.644729 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6\n 2026-03-20 16:26:25.644733 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3\n 2026-03-20 16:26:25.644737 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk\n 2026-03-20 16:26:25.644742 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb\n 2026-03-20 16:26:25.644746 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE\n 2026-03-20 16:26:25.644750 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==\n 2026-03-20 16:26:25.644754 | controller | \ logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.644758 | controller | CERTIFICATE-----' ']'\n logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.644766 | controller | | running command: [sh -c set -euxo pipefail\n public_svc_cert=$(oc rsh -n 2026-03-20 16:26:25.644770 | controller | $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n 2026-03-20 16:26:25.644774 | controller | \ public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-20 16:26:25.644778 | controller | -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc 2026-03-20 16:26:25.644782 | controller | secret and cert secret match\n if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" 2026-03-20 16:26:25.644787 | controller | ]; then\n exit 1\n fi\n \n internal_svc_cert=$(oc 2026-03-20 16:26:25.644793 | controller | rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n 2026-03-20 16:26:25.644797 | controller | \ internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-20 16:26:25.644801 | controller | -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc 2026-03-20 16:26:25.644805 | controller | secret and cert secret match\n if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" 2026-03-20 16:26:25.644810 | controller | ]; then\n exit 1\n fi\n ]\n logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.644826 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-20 16:26:25.644832 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='-----BEGIN 2026-03-20 16:26:25.644878 | controller | CERTIFICATE-----\n logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.644893 | controller | | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY\n logger.go:42: 2026-03-20 16:26:25.644898 | controller | 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx\n 2026-03-20 16:26:25.644910 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu\n 2026-03-20 16:26:25.644915 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH\n 2026-03-20 16:26:25.644919 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol\n 2026-03-20 16:26:25.644923 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X\n 2026-03-20 16:26:25.644927 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI\n 2026-03-20 16:26:25.644947 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28\n 2026-03-20 16:26:25.644954 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww\n 2026-03-20 16:26:25.644958 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/\n 2026-03-20 16:26:25.644962 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy\n 2026-03-20 16:26:25.644967 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH\n 2026-03-20 16:26:25.644971 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp\n 2026-03-20 16:26:25.644984 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6\n 2026-03-20 16:26:25.644990 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv\n 2026-03-20 16:26:25.644995 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl\n 2026-03-20 16:26:25.644999 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs\n 2026-03-20 16:26:25.645003 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0\n 2026-03-20 16:26:25.645007 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9\n 2026-03-20 16:26:25.645011 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=\n 2026-03-20 16:26:25.645016 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.645020 | controller | CERTIFICATE-----'\n logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.645024 | controller | | ++ base64 --decode\n logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.645028 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n 2026-03-20 16:26:25.645033 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert='-----BEGIN 2026-03-20 16:26:25.645037 | controller | CERTIFICATE-----\n logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.645041 | controller | | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY\n logger.go:42: 2026-03-20 16:26:25.645045 | controller | 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx\n 2026-03-20 16:26:25.645049 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu\n 2026-03-20 16:26:25.645062 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH\n 2026-03-20 16:26:25.645068 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol\n 2026-03-20 16:26:25.645072 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X\n 2026-03-20 16:26:25.645076 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI\n 2026-03-20 16:26:25.645080 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28\n 2026-03-20 16:26:25.645085 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww\n 2026-03-20 16:26:25.645089 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/\n 2026-03-20 16:26:25.645093 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy\n 2026-03-20 16:26:25.645097 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH\n 2026-03-20 16:26:25.645105 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp\n 2026-03-20 16:26:25.645112 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6\n 2026-03-20 16:26:25.645116 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv\n 2026-03-20 16:26:25.645120 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl\n 2026-03-20 16:26:25.645125 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs\n 2026-03-20 16:26:25.645129 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0\n 2026-03-20 16:26:25.645133 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9\n 2026-03-20 16:26:25.645137 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=\n 2026-03-20 16:26:25.645142 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.645146 | controller | CERTIFICATE-----'\n logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.645150 | controller | | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.645154 | controller | | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY\n logger.go:42: 2026-03-20 16:26:25.645159 | controller | 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx\n 2026-03-20 16:26:25.645163 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu\n 2026-03-20 16:26:25.645167 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH\n 2026-03-20 16:26:25.645171 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol\n 2026-03-20 16:26:25.645175 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X\n 2026-03-20 16:26:25.645180 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI\n 2026-03-20 16:26:25.645184 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28\n 2026-03-20 16:26:25.645188 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww\n 2026-03-20 16:26:25.645199 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/\n 2026-03-20 16:26:25.645203 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy\n 2026-03-20 16:26:25.645208 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH\n 2026-03-20 16:26:25.645212 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp\n 2026-03-20 16:26:25.645216 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6\n 2026-03-20 16:26:25.645223 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv\n 2026-03-20 16:26:25.645227 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl\n 2026-03-20 16:26:25.645233 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs\n 2026-03-20 16:26:25.645238 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0\n 2026-03-20 16:26:25.645242 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9\n 2026-03-20 16:26:25.645246 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=\n 2026-03-20 16:26:25.645250 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.645254 | controller | CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 16:18:58 2026-03-20 16:26:25.645259 | controller | | watcher-tls-certs-change/1-deploy-with-tlse | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY\n 2026-03-20 16:26:25.645263 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx\n 2026-03-20 16:26:25.645267 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu\n 2026-03-20 16:26:25.645271 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH\n 2026-03-20 16:26:25.645275 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol\n 2026-03-20 16:26:25.645280 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X\n 2026-03-20 16:26:25.645284 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI\n 2026-03-20 16:26:25.645288 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28\n 2026-03-20 16:26:25.645292 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww\n 2026-03-20 16:26:25.645296 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/\n 2026-03-20 16:26:25.645301 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy\n 2026-03-20 16:26:25.645305 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH\n 2026-03-20 16:26:25.645309 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp\n 2026-03-20 16:26:25.645313 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6\n 2026-03-20 16:26:25.645318 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv\n 2026-03-20 16:26:25.645322 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl\n 2026-03-20 16:26:25.645335 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs\n 2026-03-20 16:26:25.645340 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0\n 2026-03-20 16:26:25.645345 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9\n 2026-03-20 16:26:25.645351 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=\n 2026-03-20 16:26:25.645355 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.645359 | controller | CERTIFICATE-----' ']'\n logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.645364 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n 2026-03-20 16:26:25.645368 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert='-----BEGIN 2026-03-20 16:26:25.645372 | controller | CERTIFICATE-----\n logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.645376 | controller | | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-20 16:26:25.645381 | controller | 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx\n 2026-03-20 16:26:25.645385 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb\n 2026-03-20 16:26:25.645389 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2\n 2026-03-20 16:26:25.645393 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro\n 2026-03-20 16:26:25.645398 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671\n 2026-03-20 16:26:25.645402 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B\n 2026-03-20 16:26:25.645406 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ\n 2026-03-20 16:26:25.645410 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-20 16:26:25.645414 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E\n 2026-03-20 16:26:25.645418 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-20 16:26:25.645423 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J\n 2026-03-20 16:26:25.645427 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU\n 2026-03-20 16:26:25.645431 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq\n 2026-03-20 16:26:25.645435 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6\n 2026-03-20 16:26:25.645439 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3\n 2026-03-20 16:26:25.645446 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk\n 2026-03-20 16:26:25.645450 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb\n 2026-03-20 16:26:25.645454 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE\n 2026-03-20 16:26:25.645458 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==\n 2026-03-20 16:26:25.645471 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.645483 | controller | CERTIFICATE-----'\n logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.645488 | controller | | ++ base64 --decode\n logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.645492 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n 2026-03-20 16:26:25.645496 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert='-----BEGIN 2026-03-20 16:26:25.645500 | controller | CERTIFICATE-----\n logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.645505 | controller | | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-20 16:26:25.645509 | controller | 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx\n 2026-03-20 16:26:25.645513 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb\n 2026-03-20 16:26:25.645517 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2\n 2026-03-20 16:26:25.645521 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro\n 2026-03-20 16:26:25.645528 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671\n 2026-03-20 16:26:25.645532 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B\n 2026-03-20 16:26:25.645536 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ\n 2026-03-20 16:26:25.645540 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-20 16:26:25.645545 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E\n 2026-03-20 16:26:25.645549 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-20 16:26:25.645553 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J\n 2026-03-20 16:26:25.645557 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU\n 2026-03-20 16:26:25.645562 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq\n 2026-03-20 16:26:25.645566 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6\n 2026-03-20 16:26:25.645572 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3\n 2026-03-20 16:26:25.645576 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk\n 2026-03-20 16:26:25.645580 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb\n 2026-03-20 16:26:25.645584 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE\n 2026-03-20 16:26:25.645589 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==\n 2026-03-20 16:26:25.645593 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.645597 | controller | CERTIFICATE-----'\n logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.645601 | controller | | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.645606 | controller | | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-20 16:26:25.645610 | controller | 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx\n 2026-03-20 16:26:25.645614 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb\n 2026-03-20 16:26:25.645625 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2\n 2026-03-20 16:26:25.645629 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro\n 2026-03-20 16:26:25.645634 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671\n 2026-03-20 16:26:25.645638 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B\n 2026-03-20 16:26:25.645642 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ\n 2026-03-20 16:26:25.645648 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-20 16:26:25.645652 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E\n 2026-03-20 16:26:25.645657 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-20 16:26:25.645661 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J\n 2026-03-20 16:26:25.645665 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU\n 2026-03-20 16:26:25.645669 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq\n 2026-03-20 16:26:25.645673 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6\n 2026-03-20 16:26:25.645677 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3\n 2026-03-20 16:26:25.645682 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk\n 2026-03-20 16:26:25.645688 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb\n 2026-03-20 16:26:25.645708 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE\n 2026-03-20 16:26:25.645716 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==\n 2026-03-20 16:26:25.645720 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.645724 | controller | CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 16:18:58 2026-03-20 16:26:25.645728 | controller | | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw\n 2026-03-20 16:26:25.645733 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx\n 2026-03-20 16:26:25.645737 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb\n 2026-03-20 16:26:25.645741 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2\n 2026-03-20 16:26:25.645745 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro\n 2026-03-20 16:26:25.645749 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671\n 2026-03-20 16:26:25.645753 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B\n 2026-03-20 16:26:25.645758 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ\n 2026-03-20 16:26:25.645762 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-20 16:26:25.645766 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E\n 2026-03-20 16:26:25.645780 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-20 16:26:25.645785 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J\n 2026-03-20 16:26:25.645789 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU\n 2026-03-20 16:26:25.645793 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq\n 2026-03-20 16:26:25.645797 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6\n 2026-03-20 16:26:25.645801 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3\n 2026-03-20 16:26:25.645806 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk\n 2026-03-20 16:26:25.645810 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb\n 2026-03-20 16:26:25.645814 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE\n 2026-03-20 16:26:25.645820 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==\n 2026-03-20 16:26:25.645824 | controller | \ logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.645828 | controller | CERTIFICATE-----' ']'\n logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.645833 | controller | | running command: [sh -c set -euxo pipefail\n public_svc_cert=$(oc rsh -n 2026-03-20 16:26:25.645837 | controller | $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n 2026-03-20 16:26:25.645841 | controller | \ public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-20 16:26:25.645845 | controller | -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc 2026-03-20 16:26:25.645849 | controller | secret and cert secret match\n if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" 2026-03-20 16:26:25.645854 | controller | ]; then\n exit 1\n fi\n \n internal_svc_cert=$(oc 2026-03-20 16:26:25.645858 | controller | rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n 2026-03-20 16:26:25.645862 | controller | \ internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-20 16:26:25.645866 | controller | -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc 2026-03-20 16:26:25.645870 | controller | secret and cert secret match\n if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" 2026-03-20 16:26:25.645875 | controller | ]; then\n exit 1\n fi\n ]\n logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.645879 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-20 16:26:25.645883 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='-----BEGIN 2026-03-20 16:26:25.645887 | controller | CERTIFICATE-----\n logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.645891 | controller | | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY\n logger.go:42: 2026-03-20 16:26:25.645896 | controller | 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx\n 2026-03-20 16:26:25.645900 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu\n 2026-03-20 16:26:25.645906 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH\n 2026-03-20 16:26:25.645911 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol\n 2026-03-20 16:26:25.645915 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X\n 2026-03-20 16:26:25.645919 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI\n 2026-03-20 16:26:25.645931 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28\n 2026-03-20 16:26:25.645935 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww\n 2026-03-20 16:26:25.645939 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/\n 2026-03-20 16:26:25.645944 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy\n 2026-03-20 16:26:25.645950 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH\n 2026-03-20 16:26:25.645954 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp\n 2026-03-20 16:26:25.645961 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6\n 2026-03-20 16:26:25.645965 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv\n 2026-03-20 16:26:25.645970 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl\n 2026-03-20 16:26:25.645974 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs\n 2026-03-20 16:26:25.645978 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0\n 2026-03-20 16:26:25.645982 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9\n 2026-03-20 16:26:25.645986 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=\n 2026-03-20 16:26:25.645991 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.645995 | controller | CERTIFICATE-----'\n logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.645999 | controller | | ++ base64 --decode\n logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.646003 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n 2026-03-20 16:26:25.646008 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert='-----BEGIN 2026-03-20 16:26:25.646012 | controller | CERTIFICATE-----\n logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.646016 | controller | | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY\n logger.go:42: 2026-03-20 16:26:25.646020 | controller | 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx\n 2026-03-20 16:26:25.646024 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu\n 2026-03-20 16:26:25.646029 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH\n 2026-03-20 16:26:25.646033 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol\n 2026-03-20 16:26:25.646037 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X\n 2026-03-20 16:26:25.646041 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI\n 2026-03-20 16:26:25.646045 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28\n 2026-03-20 16:26:25.646050 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww\n 2026-03-20 16:26:25.646054 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/\n 2026-03-20 16:26:25.646060 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy\n 2026-03-20 16:26:25.646070 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH\n 2026-03-20 16:26:25.646075 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp\n 2026-03-20 16:26:25.646079 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6\n 2026-03-20 16:26:25.646083 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv\n 2026-03-20 16:26:25.646088 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl\n 2026-03-20 16:26:25.646092 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs\n 2026-03-20 16:26:25.646098 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0\n 2026-03-20 16:26:25.646102 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9\n 2026-03-20 16:26:25.646107 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=\n 2026-03-20 16:26:25.646111 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.646115 | controller | CERTIFICATE-----'\n logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.646119 | controller | | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.646124 | controller | | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY\n logger.go:42: 2026-03-20 16:26:25.646130 | controller | 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx\n 2026-03-20 16:26:25.646134 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu\n 2026-03-20 16:26:25.646138 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH\n 2026-03-20 16:26:25.646142 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol\n 2026-03-20 16:26:25.646146 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X\n 2026-03-20 16:26:25.646151 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI\n 2026-03-20 16:26:25.646155 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28\n 2026-03-20 16:26:25.646159 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww\n 2026-03-20 16:26:25.646163 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/\n 2026-03-20 16:26:25.646167 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy\n 2026-03-20 16:26:25.646172 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH\n 2026-03-20 16:26:25.646178 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp\n 2026-03-20 16:26:25.646182 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6\n 2026-03-20 16:26:25.646186 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv\n 2026-03-20 16:26:25.646190 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl\n 2026-03-20 16:26:25.646195 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs\n 2026-03-20 16:26:25.646199 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0\n 2026-03-20 16:26:25.646210 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9\n 2026-03-20 16:26:25.646215 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=\n 2026-03-20 16:26:25.646219 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.646223 | controller | CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 16:19:00 2026-03-20 16:26:25.646227 | controller | | watcher-tls-certs-change/1-deploy-with-tlse | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY\n 2026-03-20 16:26:25.646231 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx\n 2026-03-20 16:26:25.646236 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu\n 2026-03-20 16:26:25.646242 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH\n 2026-03-20 16:26:25.646246 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol\n 2026-03-20 16:26:25.646250 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X\n 2026-03-20 16:26:25.646255 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI\n 2026-03-20 16:26:25.646259 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28\n 2026-03-20 16:26:25.646263 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww\n 2026-03-20 16:26:25.646267 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/\n 2026-03-20 16:26:25.646271 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy\n 2026-03-20 16:26:25.646276 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH\n 2026-03-20 16:26:25.646280 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp\n 2026-03-20 16:26:25.646284 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6\n 2026-03-20 16:26:25.646291 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv\n 2026-03-20 16:26:25.646295 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl\n 2026-03-20 16:26:25.646310 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs\n 2026-03-20 16:26:25.646315 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0\n 2026-03-20 16:26:25.646319 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9\n 2026-03-20 16:26:25.646323 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=\n 2026-03-20 16:26:25.646327 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.646331 | controller | CERTIFICATE-----' ']'\n logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.646338 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n 2026-03-20 16:26:25.646342 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert='-----BEGIN 2026-03-20 16:26:25.646346 | controller | CERTIFICATE-----\n logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.646352 | controller | | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-20 16:26:25.646365 | controller | 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx\n 2026-03-20 16:26:25.646370 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb\n 2026-03-20 16:26:25.646382 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2\n 2026-03-20 16:26:25.646386 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro\n 2026-03-20 16:26:25.646391 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671\n 2026-03-20 16:26:25.646395 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B\n 2026-03-20 16:26:25.646401 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ\n 2026-03-20 16:26:25.646405 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-20 16:26:25.646409 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E\n 2026-03-20 16:26:25.646414 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-20 16:26:25.646418 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J\n 2026-03-20 16:26:25.646422 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU\n 2026-03-20 16:26:25.646429 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq\n 2026-03-20 16:26:25.646433 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6\n 2026-03-20 16:26:25.646438 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3\n 2026-03-20 16:26:25.646442 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk\n 2026-03-20 16:26:25.646454 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb\n 2026-03-20 16:26:25.646459 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE\n 2026-03-20 16:26:25.646463 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==\n 2026-03-20 16:26:25.646467 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.646471 | controller | CERTIFICATE-----'\n logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.646476 | controller | | ++ base64 --decode\n logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.646480 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n 2026-03-20 16:26:25.646484 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert='-----BEGIN 2026-03-20 16:26:25.646488 | controller | CERTIFICATE-----\n logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.646493 | controller | | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-20 16:26:25.646497 | controller | 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx\n 2026-03-20 16:26:25.646501 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb\n 2026-03-20 16:26:25.646505 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2\n 2026-03-20 16:26:25.646509 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro\n 2026-03-20 16:26:25.646527 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671\n 2026-03-20 16:26:25.646532 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B\n 2026-03-20 16:26:25.646537 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ\n 2026-03-20 16:26:25.646541 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-20 16:26:25.646545 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E\n 2026-03-20 16:26:25.646552 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-20 16:26:25.646556 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J\n 2026-03-20 16:26:25.646560 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU\n 2026-03-20 16:26:25.646566 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq\n 2026-03-20 16:26:25.646570 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6\n 2026-03-20 16:26:25.646574 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3\n 2026-03-20 16:26:25.646579 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk\n 2026-03-20 16:26:25.646583 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb\n 2026-03-20 16:26:25.646587 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE\n 2026-03-20 16:26:25.646591 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==\n 2026-03-20 16:26:25.646595 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.646600 | controller | CERTIFICATE-----'\n logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.646604 | controller | | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.646608 | controller | | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-20 16:26:25.646612 | controller | 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx\n 2026-03-20 16:26:25.646616 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb\n 2026-03-20 16:26:25.646621 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2\n 2026-03-20 16:26:25.646625 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro\n 2026-03-20 16:26:25.646629 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671\n 2026-03-20 16:26:25.646633 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B\n 2026-03-20 16:26:25.646637 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ\n 2026-03-20 16:26:25.646642 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-20 16:26:25.646646 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E\n 2026-03-20 16:26:25.646650 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-20 16:26:25.646654 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J\n 2026-03-20 16:26:25.646665 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU\n 2026-03-20 16:26:25.646672 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq\n 2026-03-20 16:26:25.646679 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6\n 2026-03-20 16:26:25.646684 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3\n 2026-03-20 16:26:25.646688 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk\n 2026-03-20 16:26:25.646705 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb\n 2026-03-20 16:26:25.646711 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE\n 2026-03-20 16:26:25.646715 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==\n 2026-03-20 16:26:25.646720 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.646724 | controller | CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 16:19:00 2026-03-20 16:26:25.646728 | controller | | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw\n 2026-03-20 16:26:25.646732 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx\n 2026-03-20 16:26:25.646737 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb\n 2026-03-20 16:26:25.646741 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2\n 2026-03-20 16:26:25.646745 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro\n 2026-03-20 16:26:25.646749 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671\n 2026-03-20 16:26:25.646753 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B\n 2026-03-20 16:26:25.646758 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ\n 2026-03-20 16:26:25.646762 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-20 16:26:25.646766 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E\n 2026-03-20 16:26:25.646770 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-20 16:26:25.646774 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J\n 2026-03-20 16:26:25.646778 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU\n 2026-03-20 16:26:25.646783 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq\n 2026-03-20 16:26:25.646787 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6\n 2026-03-20 16:26:25.646793 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3\n 2026-03-20 16:26:25.646800 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk\n 2026-03-20 16:26:25.646804 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb\n 2026-03-20 16:26:25.646815 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE\n 2026-03-20 16:26:25.646820 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==\n 2026-03-20 16:26:25.646824 | controller | \ logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.646828 | controller | CERTIFICATE-----' ']'\n logger.go:42: 16:19:01 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.646832 | controller | | running command: [sh -c set -euxo pipefail\n public_svc_cert=$(oc rsh -n 2026-03-20 16:26:25.646837 | controller | $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n 2026-03-20 16:26:25.646841 | controller | \ public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-20 16:26:25.646845 | controller | -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc 2026-03-20 16:26:25.646849 | controller | secret and cert secret match\n if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" 2026-03-20 16:26:25.646853 | controller | ]; then\n exit 1\n fi\n \n internal_svc_cert=$(oc 2026-03-20 16:26:25.646858 | controller | rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n 2026-03-20 16:26:25.646862 | controller | \ internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-20 16:26:25.646866 | controller | -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc 2026-03-20 16:26:25.646870 | controller | secret and cert secret match\n if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" 2026-03-20 16:26:25.646874 | controller | ]; then\n exit 1\n fi\n ]\n logger.go:42: 16:19:01 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.646879 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-20 16:26:25.646883 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='-----BEGIN 2026-03-20 16:26:25.646887 | controller | CERTIFICATE-----\n logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.646891 | controller | | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY\n logger.go:42: 2026-03-20 16:26:25.646896 | controller | 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx\n 2026-03-20 16:26:25.646900 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu\n 2026-03-20 16:26:25.646905 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH\n 2026-03-20 16:26:25.646909 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol\n 2026-03-20 16:26:25.646913 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X\n 2026-03-20 16:26:25.646918 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI\n 2026-03-20 16:26:25.646922 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28\n 2026-03-20 16:26:25.646937 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww\n 2026-03-20 16:26:25.646942 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/\n 2026-03-20 16:26:25.646946 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy\n 2026-03-20 16:26:25.646950 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH\n 2026-03-20 16:26:25.646954 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp\n 2026-03-20 16:26:25.646958 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6\n 2026-03-20 16:26:25.646963 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv\n 2026-03-20 16:26:25.646967 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl\n 2026-03-20 16:26:25.646978 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs\n 2026-03-20 16:26:25.646983 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0\n 2026-03-20 16:26:25.646990 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9\n 2026-03-20 16:26:25.646994 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=\n 2026-03-20 16:26:25.646999 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.647003 | controller | CERTIFICATE-----'\n logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.647007 | controller | | ++ base64 --decode\n logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.647011 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n 2026-03-20 16:26:25.647015 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert='-----BEGIN 2026-03-20 16:26:25.647020 | controller | CERTIFICATE-----\n logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.647024 | controller | | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY\n logger.go:42: 2026-03-20 16:26:25.647028 | controller | 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx\n 2026-03-20 16:26:25.647032 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu\n 2026-03-20 16:26:25.647036 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH\n 2026-03-20 16:26:25.647041 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol\n 2026-03-20 16:26:25.647045 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X\n 2026-03-20 16:26:25.647049 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI\n 2026-03-20 16:26:25.647056 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28\n 2026-03-20 16:26:25.647061 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww\n 2026-03-20 16:26:25.647065 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/\n 2026-03-20 16:26:25.647069 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy\n 2026-03-20 16:26:25.647073 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH\n 2026-03-20 16:26:25.647077 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp\n 2026-03-20 16:26:25.647082 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6\n 2026-03-20 16:26:25.647086 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv\n 2026-03-20 16:26:25.647090 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl\n 2026-03-20 16:26:25.647094 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs\n 2026-03-20 16:26:25.647098 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0\n 2026-03-20 16:26:25.647103 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9\n 2026-03-20 16:26:25.647116 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=\n 2026-03-20 16:26:25.647120 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.647131 | controller | CERTIFICATE-----'\n logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.647136 | controller | | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.647140 | controller | | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY\n logger.go:42: 2026-03-20 16:26:25.647144 | controller | 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx\n 2026-03-20 16:26:25.647148 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu\n 2026-03-20 16:26:25.647153 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH\n 2026-03-20 16:26:25.647157 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol\n 2026-03-20 16:26:25.647163 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X\n 2026-03-20 16:26:25.647168 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI\n 2026-03-20 16:26:25.647172 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28\n 2026-03-20 16:26:25.647176 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww\n 2026-03-20 16:26:25.647191 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/\n 2026-03-20 16:26:25.647195 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy\n 2026-03-20 16:26:25.647199 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH\n 2026-03-20 16:26:25.647204 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp\n 2026-03-20 16:26:25.647208 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6\n 2026-03-20 16:26:25.647212 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv\n 2026-03-20 16:26:25.647216 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl\n 2026-03-20 16:26:25.647221 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs\n 2026-03-20 16:26:25.647225 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0\n 2026-03-20 16:26:25.647229 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9\n 2026-03-20 16:26:25.647233 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=\n 2026-03-20 16:26:25.647237 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.647242 | controller | CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 16:19:02 2026-03-20 16:26:25.647246 | controller | | watcher-tls-certs-change/1-deploy-with-tlse | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY\n 2026-03-20 16:26:25.647250 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx\n 2026-03-20 16:26:25.647254 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu\n 2026-03-20 16:26:25.647258 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH\n 2026-03-20 16:26:25.647263 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol\n 2026-03-20 16:26:25.647273 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X\n 2026-03-20 16:26:25.647277 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI\n 2026-03-20 16:26:25.647282 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28\n 2026-03-20 16:26:25.647295 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww\n 2026-03-20 16:26:25.647299 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/\n 2026-03-20 16:26:25.647303 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy\n 2026-03-20 16:26:25.647309 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH\n 2026-03-20 16:26:25.647322 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp\n 2026-03-20 16:26:25.647326 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6\n 2026-03-20 16:26:25.647331 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv\n 2026-03-20 16:26:25.647335 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl\n 2026-03-20 16:26:25.647339 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs\n 2026-03-20 16:26:25.647343 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0\n 2026-03-20 16:26:25.647347 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9\n 2026-03-20 16:26:25.647352 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=\n 2026-03-20 16:26:25.647356 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.647360 | controller | CERTIFICATE-----' ']'\n logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.647364 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n 2026-03-20 16:26:25.647369 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert='-----BEGIN 2026-03-20 16:26:25.647373 | controller | CERTIFICATE-----\n logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.647377 | controller | | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-20 16:26:25.647381 | controller | 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx\n 2026-03-20 16:26:25.647385 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb\n 2026-03-20 16:26:25.647390 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2\n 2026-03-20 16:26:25.647394 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro\n 2026-03-20 16:26:25.647398 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671\n 2026-03-20 16:26:25.647402 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B\n 2026-03-20 16:26:25.647407 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ\n 2026-03-20 16:26:25.647411 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-20 16:26:25.647415 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E\n 2026-03-20 16:26:25.647425 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-20 16:26:25.647432 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J\n 2026-03-20 16:26:25.647438 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU\n 2026-03-20 16:26:25.647442 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq\n 2026-03-20 16:26:25.647446 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6\n 2026-03-20 16:26:25.647450 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3\n 2026-03-20 16:26:25.647455 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk\n 2026-03-20 16:26:25.647459 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb\n 2026-03-20 16:26:25.647463 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE\n 2026-03-20 16:26:25.647467 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==\n 2026-03-20 16:26:25.647472 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.647476 | controller | CERTIFICATE-----'\n logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.647480 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n 2026-03-20 16:26:25.647484 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 2026-03-20 16:26:25.647489 | controller | --decode\n logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.647493 | controller | | + internal_secret_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 16:19:02 2026-03-20 16:26:25.647497 | controller | | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw\n 2026-03-20 16:26:25.647501 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx\n 2026-03-20 16:26:25.647506 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb\n 2026-03-20 16:26:25.647510 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2\n 2026-03-20 16:26:25.647514 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro\n 2026-03-20 16:26:25.647518 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671\n 2026-03-20 16:26:25.647522 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B\n 2026-03-20 16:26:25.647526 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ\n 2026-03-20 16:26:25.647531 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-20 16:26:25.647535 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E\n 2026-03-20 16:26:25.647558 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-20 16:26:25.647564 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J\n 2026-03-20 16:26:25.647569 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU\n 2026-03-20 16:26:25.647574 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq\n 2026-03-20 16:26:25.647593 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6\n 2026-03-20 16:26:25.647599 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3\n 2026-03-20 16:26:25.647604 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk\n 2026-03-20 16:26:25.647609 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb\n 2026-03-20 16:26:25.647614 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE\n 2026-03-20 16:26:25.647619 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==\n 2026-03-20 16:26:25.647624 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.647629 | controller | CERTIFICATE-----'\n logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.647633 | controller | | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.647638 | controller | | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-20 16:26:25.647643 | controller | 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx\n 2026-03-20 16:26:25.647648 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb\n 2026-03-20 16:26:25.647653 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2\n 2026-03-20 16:26:25.647658 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro\n 2026-03-20 16:26:25.647663 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671\n 2026-03-20 16:26:25.647668 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B\n 2026-03-20 16:26:25.647673 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ\n 2026-03-20 16:26:25.647678 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-20 16:26:25.647683 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E\n 2026-03-20 16:26:25.647688 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-20 16:26:25.647708 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J\n 2026-03-20 16:26:25.647715 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU\n 2026-03-20 16:26:25.647720 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq\n 2026-03-20 16:26:25.647725 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6\n 2026-03-20 16:26:25.647730 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3\n 2026-03-20 16:26:25.647735 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk\n 2026-03-20 16:26:25.647744 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb\n 2026-03-20 16:26:25.647749 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE\n 2026-03-20 16:26:25.647754 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==\n 2026-03-20 16:26:25.647759 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.647773 | controller | CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 16:19:02 2026-03-20 16:26:25.647779 | controller | | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw\n 2026-03-20 16:26:25.647784 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx\n 2026-03-20 16:26:25.647790 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb\n 2026-03-20 16:26:25.647796 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2\n 2026-03-20 16:26:25.647802 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro\n 2026-03-20 16:26:25.647808 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671\n 2026-03-20 16:26:25.647814 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B\n 2026-03-20 16:26:25.647820 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ\n 2026-03-20 16:26:25.647825 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-20 16:26:25.647831 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E\n 2026-03-20 16:26:25.647837 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-20 16:26:25.647842 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J\n 2026-03-20 16:26:25.647846 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU\n 2026-03-20 16:26:25.647853 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq\n 2026-03-20 16:26:25.647857 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6\n 2026-03-20 16:26:25.647861 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3\n 2026-03-20 16:26:25.647865 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk\n 2026-03-20 16:26:25.647869 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb\n 2026-03-20 16:26:25.647874 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE\n 2026-03-20 16:26:25.647880 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==\n 2026-03-20 16:26:25.647884 | controller | \ logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.647889 | controller | CERTIFICATE-----' ']'\n logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.647893 | controller | | running command: [sh -c set -euxo pipefail\n public_svc_cert=$(oc rsh -n 2026-03-20 16:26:25.647897 | controller | $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n 2026-03-20 16:26:25.647901 | controller | \ public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-20 16:26:25.647905 | controller | -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc 2026-03-20 16:26:25.647910 | controller | secret and cert secret match\n if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" 2026-03-20 16:26:25.647914 | controller | ]; then\n exit 1\n fi\n \n internal_svc_cert=$(oc 2026-03-20 16:26:25.647918 | controller | rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n 2026-03-20 16:26:25.647922 | controller | \ internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-20 16:26:25.647926 | controller | -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc 2026-03-20 16:26:25.647937 | controller | secret and cert secret match\n if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" 2026-03-20 16:26:25.647943 | controller | ]; then\n exit 1\n fi\n ]\n logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.647947 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-20 16:26:25.647951 | controller | \ logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='-----BEGIN 2026-03-20 16:26:25.647956 | controller | CERTIFICATE-----\n logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.647960 | controller | | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY\n logger.go:42: 2026-03-20 16:26:25.647964 | controller | 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx\n 2026-03-20 16:26:25.647968 | controller | \ logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu\n 2026-03-20 16:26:25.647972 | controller | \ logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH\n 2026-03-20 16:26:25.647977 | controller | \ logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol\n 2026-03-20 16:26:25.647982 | controller | \ logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X\n 2026-03-20 16:26:25.647987 | controller | \ logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI\n 2026-03-20 16:26:25.647991 | controller | \ logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28\n 2026-03-20 16:26:25.647995 | controller | \ logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww\n 2026-03-20 16:26:25.647999 | controller | \ logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/\n 2026-03-20 16:26:25.648004 | controller | \ logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy\n 2026-03-20 16:26:25.648008 | controller | \ logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH\n 2026-03-20 16:26:25.648012 | controller | \ logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp\n 2026-03-20 16:26:25.648016 | controller | \ logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6\n 2026-03-20 16:26:25.648020 | controller | \ logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv\n 2026-03-20 16:26:25.648024 | controller | \ logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl\n 2026-03-20 16:26:25.648029 | controller | \ logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs\n 2026-03-20 16:26:25.648033 | controller | \ logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0\n 2026-03-20 16:26:25.648037 | controller | \ logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9\n 2026-03-20 16:26:25.648041 | controller | \ logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=\n 2026-03-20 16:26:25.648045 | controller | \ logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.648053 | controller | CERTIFICATE-----'\n logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.648057 | controller | | ++ base64 --decode\n logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.648061 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n 2026-03-20 16:26:25.648066 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert='-----BEGIN 2026-03-20 16:26:25.648070 | controller | CERTIFICATE-----\n logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.648074 | controller | | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY\n logger.go:42: 2026-03-20 16:26:25.648084 | controller | 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx\n 2026-03-20 16:26:25.648088 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu\n 2026-03-20 16:26:25.648093 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH\n 2026-03-20 16:26:25.648099 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol\n 2026-03-20 16:26:25.648106 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X\n 2026-03-20 16:26:25.648110 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI\n 2026-03-20 16:26:25.648114 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28\n 2026-03-20 16:26:25.648119 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww\n 2026-03-20 16:26:25.648123 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/\n 2026-03-20 16:26:25.648127 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy\n 2026-03-20 16:26:25.648131 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH\n 2026-03-20 16:26:25.648135 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp\n 2026-03-20 16:26:25.648140 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6\n 2026-03-20 16:26:25.648144 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv\n 2026-03-20 16:26:25.648148 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl\n 2026-03-20 16:26:25.648152 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs\n 2026-03-20 16:26:25.648156 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0\n 2026-03-20 16:26:25.648171 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9\n 2026-03-20 16:26:25.648175 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=\n 2026-03-20 16:26:25.648179 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.648183 | controller | CERTIFICATE-----'\n logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.648187 | controller | | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.648192 | controller | | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY\n logger.go:42: 2026-03-20 16:26:25.648196 | controller | 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx\n 2026-03-20 16:26:25.648200 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu\n 2026-03-20 16:26:25.648204 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH\n 2026-03-20 16:26:25.648209 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol\n 2026-03-20 16:26:25.648215 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X\n 2026-03-20 16:26:25.648223 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI\n 2026-03-20 16:26:25.648242 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28\n 2026-03-20 16:26:25.648247 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww\n 2026-03-20 16:26:25.648263 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/\n 2026-03-20 16:26:25.648268 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy\n 2026-03-20 16:26:25.648272 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH\n 2026-03-20 16:26:25.648276 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp\n 2026-03-20 16:26:25.648280 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6\n 2026-03-20 16:26:25.648284 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv\n 2026-03-20 16:26:25.648289 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl\n 2026-03-20 16:26:25.648293 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs\n 2026-03-20 16:26:25.648297 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0\n 2026-03-20 16:26:25.648301 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9\n 2026-03-20 16:26:25.648305 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=\n 2026-03-20 16:26:25.648310 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.648322 | controller | CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 16:19:04 2026-03-20 16:26:25.648326 | controller | | watcher-tls-certs-change/1-deploy-with-tlse | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY\n 2026-03-20 16:26:25.648331 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx\n 2026-03-20 16:26:25.648335 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu\n 2026-03-20 16:26:25.648339 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH\n 2026-03-20 16:26:25.648343 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol\n 2026-03-20 16:26:25.648347 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X\n 2026-03-20 16:26:25.648352 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI\n 2026-03-20 16:26:25.648356 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28\n 2026-03-20 16:26:25.648362 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww\n 2026-03-20 16:26:25.648367 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/\n 2026-03-20 16:26:25.648371 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy\n 2026-03-20 16:26:25.648375 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH\n 2026-03-20 16:26:25.648382 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp\n 2026-03-20 16:26:25.648386 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6\n 2026-03-20 16:26:25.648397 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv\n 2026-03-20 16:26:25.648402 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl\n 2026-03-20 16:26:25.648406 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs\n 2026-03-20 16:26:25.648410 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0\n 2026-03-20 16:26:25.648415 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9\n 2026-03-20 16:26:25.648419 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=\n 2026-03-20 16:26:25.648423 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.648427 | controller | CERTIFICATE-----' ']'\n logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.648431 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n 2026-03-20 16:26:25.648436 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert='-----BEGIN 2026-03-20 16:26:25.648440 | controller | CERTIFICATE-----\n logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.648444 | controller | | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-20 16:26:25.648448 | controller | 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx\n 2026-03-20 16:26:25.648453 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb\n 2026-03-20 16:26:25.648457 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2\n 2026-03-20 16:26:25.648461 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro\n 2026-03-20 16:26:25.648465 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671\n 2026-03-20 16:26:25.648469 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B\n 2026-03-20 16:26:25.648474 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ\n 2026-03-20 16:26:25.648479 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-20 16:26:25.648484 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E\n 2026-03-20 16:26:25.648488 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-20 16:26:25.648492 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J\n 2026-03-20 16:26:25.648496 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU\n 2026-03-20 16:26:25.648503 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq\n 2026-03-20 16:26:25.648507 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6\n 2026-03-20 16:26:25.648512 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3\n 2026-03-20 16:26:25.648518 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk\n 2026-03-20 16:26:25.648522 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb\n 2026-03-20 16:26:25.648526 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE\n 2026-03-20 16:26:25.648536 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==\n 2026-03-20 16:26:25.648540 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.648545 | controller | CERTIFICATE-----'\n logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.648549 | controller | | ++ base64 --decode\n logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.648553 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n 2026-03-20 16:26:25.648557 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert='-----BEGIN 2026-03-20 16:26:25.648562 | controller | CERTIFICATE-----\n logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.648566 | controller | | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-20 16:26:25.648570 | controller | 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx\n 2026-03-20 16:26:25.648574 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb\n 2026-03-20 16:26:25.648578 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2\n 2026-03-20 16:26:25.648583 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro\n 2026-03-20 16:26:25.648587 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671\n 2026-03-20 16:26:25.648591 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B\n 2026-03-20 16:26:25.648600 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ\n 2026-03-20 16:26:25.648604 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-20 16:26:25.648608 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E\n 2026-03-20 16:26:25.648613 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-20 16:26:25.648617 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J\n 2026-03-20 16:26:25.648621 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU\n 2026-03-20 16:26:25.648625 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq\n 2026-03-20 16:26:25.648629 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6\n 2026-03-20 16:26:25.648633 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3\n 2026-03-20 16:26:25.648638 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk\n 2026-03-20 16:26:25.648642 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb\n 2026-03-20 16:26:25.648646 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE\n 2026-03-20 16:26:25.648659 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==\n 2026-03-20 16:26:25.648664 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.648668 | controller | CERTIFICATE-----'\n logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.648672 | controller | | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.648677 | controller | | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-20 16:26:25.648688 | controller | 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx\n 2026-03-20 16:26:25.648705 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb\n 2026-03-20 16:26:25.648713 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2\n 2026-03-20 16:26:25.648717 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro\n 2026-03-20 16:26:25.648721 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671\n 2026-03-20 16:26:25.648725 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B\n 2026-03-20 16:26:25.648729 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ\n 2026-03-20 16:26:25.648735 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-20 16:26:25.648740 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E\n 2026-03-20 16:26:25.648744 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-20 16:26:25.648748 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J\n 2026-03-20 16:26:25.648752 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU\n 2026-03-20 16:26:25.648756 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq\n 2026-03-20 16:26:25.648761 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6\n 2026-03-20 16:26:25.648765 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3\n 2026-03-20 16:26:25.648777 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk\n 2026-03-20 16:26:25.648781 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb\n 2026-03-20 16:26:25.648785 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE\n 2026-03-20 16:26:25.648790 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==\n 2026-03-20 16:26:25.648794 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.648798 | controller | CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 16:19:04 2026-03-20 16:26:25.648802 | controller | | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw\n 2026-03-20 16:26:25.648806 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx\n 2026-03-20 16:26:25.648811 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb\n 2026-03-20 16:26:25.648818 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2\n 2026-03-20 16:26:25.648822 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro\n 2026-03-20 16:26:25.648827 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671\n 2026-03-20 16:26:25.648831 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B\n 2026-03-20 16:26:25.648835 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ\n 2026-03-20 16:26:25.648846 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-20 16:26:25.648850 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E\n 2026-03-20 16:26:25.648856 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-20 16:26:25.648861 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J\n 2026-03-20 16:26:25.648865 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU\n 2026-03-20 16:26:25.648869 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq\n 2026-03-20 16:26:25.648873 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6\n 2026-03-20 16:26:25.648878 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3\n 2026-03-20 16:26:25.648882 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk\n 2026-03-20 16:26:25.648886 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb\n 2026-03-20 16:26:25.648890 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE\n 2026-03-20 16:26:25.648894 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==\n 2026-03-20 16:26:25.648898 | controller | \ logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.648903 | controller | CERTIFICATE-----' ']'\n logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.648907 | controller | | running command: [sh -c set -euxo pipefail\n public_svc_cert=$(oc rsh -n 2026-03-20 16:26:25.648911 | controller | $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n 2026-03-20 16:26:25.648915 | controller | \ public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-20 16:26:25.648920 | controller | -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc 2026-03-20 16:26:25.648924 | controller | secret and cert secret match\n if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" 2026-03-20 16:26:25.648928 | controller | ]; then\n exit 1\n fi\n \n internal_svc_cert=$(oc 2026-03-20 16:26:25.648932 | controller | rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n 2026-03-20 16:26:25.648937 | controller | \ internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-20 16:26:25.648942 | controller | -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc 2026-03-20 16:26:25.648947 | controller | secret and cert secret match\n if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" 2026-03-20 16:26:25.648951 | controller | ]; then\n exit 1\n fi\n ]\n logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.648955 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-20 16:26:25.648959 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='-----BEGIN 2026-03-20 16:26:25.648964 | controller | CERTIFICATE-----\n logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.648969 | controller | | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY\n logger.go:42: 2026-03-20 16:26:25.648974 | controller | 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx\n 2026-03-20 16:26:25.648979 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu\n 2026-03-20 16:26:25.648984 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH\n 2026-03-20 16:26:25.648988 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol\n 2026-03-20 16:26:25.648992 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X\n 2026-03-20 16:26:25.649005 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI\n 2026-03-20 16:26:25.649010 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28\n 2026-03-20 16:26:25.649014 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww\n 2026-03-20 16:26:25.649018 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/\n 2026-03-20 16:26:25.649023 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy\n 2026-03-20 16:26:25.649027 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH\n 2026-03-20 16:26:25.649031 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp\n 2026-03-20 16:26:25.649035 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6\n 2026-03-20 16:26:25.649039 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv\n 2026-03-20 16:26:25.649043 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl\n 2026-03-20 16:26:25.649048 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs\n 2026-03-20 16:26:25.649052 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0\n 2026-03-20 16:26:25.649056 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9\n 2026-03-20 16:26:25.649060 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=\n 2026-03-20 16:26:25.649064 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.649069 | controller | CERTIFICATE-----'\n logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.649073 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n 2026-03-20 16:26:25.649077 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 2026-03-20 16:26:25.649081 | controller | --decode\n logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.649085 | controller | | + public_secret_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 16:19:05 2026-03-20 16:26:25.649090 | controller | | watcher-tls-certs-change/1-deploy-with-tlse | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY\n 2026-03-20 16:26:25.649094 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx\n 2026-03-20 16:26:25.649100 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu\n 2026-03-20 16:26:25.649104 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH\n 2026-03-20 16:26:25.649108 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol\n 2026-03-20 16:26:25.649113 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X\n 2026-03-20 16:26:25.649117 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI\n 2026-03-20 16:26:25.649121 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28\n 2026-03-20 16:26:25.649125 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww\n 2026-03-20 16:26:25.649132 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/\n 2026-03-20 16:26:25.649141 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy\n 2026-03-20 16:26:25.649146 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH\n 2026-03-20 16:26:25.649150 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp\n 2026-03-20 16:26:25.649154 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6\n 2026-03-20 16:26:25.649159 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv\n 2026-03-20 16:26:25.649163 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl\n 2026-03-20 16:26:25.649167 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs\n 2026-03-20 16:26:25.649171 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0\n 2026-03-20 16:26:25.649175 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9\n 2026-03-20 16:26:25.649180 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=\n 2026-03-20 16:26:25.649184 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.649188 | controller | CERTIFICATE-----'\n logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.649192 | controller | | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.649197 | controller | | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY\n logger.go:42: 2026-03-20 16:26:25.649201 | controller | 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx\n 2026-03-20 16:26:25.649205 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu\n 2026-03-20 16:26:25.649212 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH\n 2026-03-20 16:26:25.649216 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol\n 2026-03-20 16:26:25.649220 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X\n 2026-03-20 16:26:25.649225 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI\n 2026-03-20 16:26:25.649229 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28\n 2026-03-20 16:26:25.649233 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww\n 2026-03-20 16:26:25.649237 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/\n 2026-03-20 16:26:25.649241 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy\n 2026-03-20 16:26:25.649245 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH\n 2026-03-20 16:26:25.649250 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp\n 2026-03-20 16:26:25.649256 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6\n 2026-03-20 16:26:25.649260 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv\n 2026-03-20 16:26:25.649264 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl\n 2026-03-20 16:26:25.649275 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs\n 2026-03-20 16:26:25.649280 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0\n 2026-03-20 16:26:25.649284 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9\n 2026-03-20 16:26:25.649289 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=\n 2026-03-20 16:26:25.649293 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.649297 | controller | CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 16:19:05 2026-03-20 16:26:25.649301 | controller | | watcher-tls-certs-change/1-deploy-with-tlse | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY\n 2026-03-20 16:26:25.649305 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx\n 2026-03-20 16:26:25.649310 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu\n 2026-03-20 16:26:25.649314 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH\n 2026-03-20 16:26:25.649318 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol\n 2026-03-20 16:26:25.649324 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X\n 2026-03-20 16:26:25.649328 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI\n 2026-03-20 16:26:25.649332 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28\n 2026-03-20 16:26:25.649337 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww\n 2026-03-20 16:26:25.649341 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/\n 2026-03-20 16:26:25.649345 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy\n 2026-03-20 16:26:25.649349 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH\n 2026-03-20 16:26:25.649353 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp\n 2026-03-20 16:26:25.649358 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6\n 2026-03-20 16:26:25.649362 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv\n 2026-03-20 16:26:25.649366 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl\n 2026-03-20 16:26:25.649374 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs\n 2026-03-20 16:26:25.649378 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0\n 2026-03-20 16:26:25.649382 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9\n 2026-03-20 16:26:25.649386 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=\n 2026-03-20 16:26:25.649391 | controller | \ logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.649395 | controller | CERTIFICATE-----' ']'\n logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.649399 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n 2026-03-20 16:26:25.649403 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert='-----BEGIN 2026-03-20 16:26:25.649414 | controller | CERTIFICATE-----\n logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.649419 | controller | | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-20 16:26:25.649423 | controller | 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx\n 2026-03-20 16:26:25.649427 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb\n 2026-03-20 16:26:25.649432 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2\n 2026-03-20 16:26:25.649436 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro\n 2026-03-20 16:26:25.649442 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671\n 2026-03-20 16:26:25.649446 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B\n 2026-03-20 16:26:25.649450 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ\n 2026-03-20 16:26:25.649454 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-20 16:26:25.649459 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E\n 2026-03-20 16:26:25.649463 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-20 16:26:25.649467 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J\n 2026-03-20 16:26:25.649471 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU\n 2026-03-20 16:26:25.649475 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq\n 2026-03-20 16:26:25.649479 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6\n 2026-03-20 16:26:25.649484 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3\n 2026-03-20 16:26:25.649488 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk\n 2026-03-20 16:26:25.649492 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb\n 2026-03-20 16:26:25.649496 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE\n 2026-03-20 16:26:25.649500 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==\n 2026-03-20 16:26:25.649514 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.649519 | controller | CERTIFICATE-----'\n logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.649523 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n 2026-03-20 16:26:25.649527 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 2026-03-20 16:26:25.649532 | controller | --decode\n logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.649536 | controller | | + internal_secret_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 16:19:06 2026-03-20 16:26:25.649540 | controller | | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw\n 2026-03-20 16:26:25.649544 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx\n 2026-03-20 16:26:25.649549 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb\n 2026-03-20 16:26:25.649553 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2\n 2026-03-20 16:26:25.649565 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro\n 2026-03-20 16:26:25.649571 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671\n 2026-03-20 16:26:25.649575 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B\n 2026-03-20 16:26:25.649579 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ\n 2026-03-20 16:26:25.649583 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-20 16:26:25.649588 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E\n 2026-03-20 16:26:25.649592 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-20 16:26:25.649596 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J\n 2026-03-20 16:26:25.649600 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU\n 2026-03-20 16:26:25.649604 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq\n 2026-03-20 16:26:25.649608 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6\n 2026-03-20 16:26:25.649613 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3\n 2026-03-20 16:26:25.649617 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk\n 2026-03-20 16:26:25.649621 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb\n 2026-03-20 16:26:25.649625 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE\n 2026-03-20 16:26:25.649642 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==\n 2026-03-20 16:26:25.649646 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.649651 | controller | CERTIFICATE-----'\n logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.649655 | controller | | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.649659 | controller | | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-20 16:26:25.649663 | controller | 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx\n 2026-03-20 16:26:25.649667 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb\n 2026-03-20 16:26:25.649672 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2\n 2026-03-20 16:26:25.649679 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro\n 2026-03-20 16:26:25.649713 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671\n 2026-03-20 16:26:25.649720 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B\n 2026-03-20 16:26:25.649725 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ\n 2026-03-20 16:26:25.649729 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-20 16:26:25.649733 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E\n 2026-03-20 16:26:25.649745 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-20 16:26:25.649749 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J\n 2026-03-20 16:26:25.649754 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU\n 2026-03-20 16:26:25.649758 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq\n 2026-03-20 16:26:25.649762 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6\n 2026-03-20 16:26:25.649766 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3\n 2026-03-20 16:26:25.649779 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk\n 2026-03-20 16:26:25.649783 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb\n 2026-03-20 16:26:25.649788 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE\n 2026-03-20 16:26:25.649792 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==\n 2026-03-20 16:26:25.649796 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.649800 | controller | CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 16:19:06 2026-03-20 16:26:25.649804 | controller | | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw\n 2026-03-20 16:26:25.649809 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx\n 2026-03-20 16:26:25.649813 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb\n 2026-03-20 16:26:25.649817 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2\n 2026-03-20 16:26:25.649821 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro\n 2026-03-20 16:26:25.649825 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671\n 2026-03-20 16:26:25.649832 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B\n 2026-03-20 16:26:25.649839 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ\n 2026-03-20 16:26:25.649843 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-20 16:26:25.649847 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E\n 2026-03-20 16:26:25.649851 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-20 16:26:25.649855 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J\n 2026-03-20 16:26:25.649859 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU\n 2026-03-20 16:26:25.649864 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq\n 2026-03-20 16:26:25.649868 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6\n 2026-03-20 16:26:25.649872 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3\n 2026-03-20 16:26:25.649876 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk\n 2026-03-20 16:26:25.649887 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb\n 2026-03-20 16:26:25.649892 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE\n 2026-03-20 16:26:25.649896 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==\n 2026-03-20 16:26:25.649900 | controller | \ logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.649905 | controller | CERTIFICATE-----' ']'\n logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.649909 | controller | | running command: [sh -c set -euxo pipefail\n public_svc_cert=$(oc rsh -n 2026-03-20 16:26:25.649913 | controller | $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n 2026-03-20 16:26:25.649917 | controller | \ public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-20 16:26:25.649922 | controller | -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc 2026-03-20 16:26:25.649926 | controller | secret and cert secret match\n if [ \"${public_svc_cert}\" != \"${public_secret_cert}\" 2026-03-20 16:26:25.649930 | controller | ]; then\n exit 1\n fi\n \n internal_svc_cert=$(oc 2026-03-20 16:26:25.649936 | controller | rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n 2026-03-20 16:26:25.649941 | controller | \ internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-20 16:26:25.649945 | controller | -o jsonpath='{.data.tls\\.crt}' | base64 --decode)\n # ensure that the svc 2026-03-20 16:26:25.649949 | controller | secret and cert secret match\n if [ \"${internal_svc_cert}\" != \"${internal_secret_cert}\" 2026-03-20 16:26:25.649953 | controller | ]; then\n exit 1\n fi\n ]\n logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.649957 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-20 16:26:25.649962 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert='-----BEGIN 2026-03-20 16:26:25.649969 | controller | CERTIFICATE-----\n logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.649973 | controller | | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY\n logger.go:42: 2026-03-20 16:26:25.649977 | controller | 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx\n 2026-03-20 16:26:25.649981 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu\n 2026-03-20 16:26:25.649985 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH\n 2026-03-20 16:26:25.649990 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol\n 2026-03-20 16:26:25.649994 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X\n 2026-03-20 16:26:25.649998 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI\n 2026-03-20 16:26:25.650002 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28\n 2026-03-20 16:26:25.650006 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww\n 2026-03-20 16:26:25.650012 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/\n 2026-03-20 16:26:25.650017 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy\n 2026-03-20 16:26:25.650021 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH\n 2026-03-20 16:26:25.650025 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp\n 2026-03-20 16:26:25.650029 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6\n 2026-03-20 16:26:25.650040 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv\n 2026-03-20 16:26:25.650045 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl\n 2026-03-20 16:26:25.650049 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs\n 2026-03-20 16:26:25.650053 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0\n 2026-03-20 16:26:25.650057 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9\n 2026-03-20 16:26:25.650061 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=\n 2026-03-20 16:26:25.650066 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.650070 | controller | CERTIFICATE-----'\n logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.650074 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n 2026-03-20 16:26:25.650078 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 2026-03-20 16:26:25.650087 | controller | --decode\n logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.650091 | controller | | + public_secret_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 16:19:07 2026-03-20 16:26:25.650096 | controller | | watcher-tls-certs-change/1-deploy-with-tlse | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY\n 2026-03-20 16:26:25.650100 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx\n 2026-03-20 16:26:25.650104 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu\n 2026-03-20 16:26:25.650108 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH\n 2026-03-20 16:26:25.650112 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol\n 2026-03-20 16:26:25.650117 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X\n 2026-03-20 16:26:25.650121 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI\n 2026-03-20 16:26:25.650125 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28\n 2026-03-20 16:26:25.650129 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww\n 2026-03-20 16:26:25.650133 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/\n 2026-03-20 16:26:25.650138 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy\n 2026-03-20 16:26:25.650142 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH\n 2026-03-20 16:26:25.650148 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp\n 2026-03-20 16:26:25.650152 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6\n 2026-03-20 16:26:25.650156 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv\n 2026-03-20 16:26:25.650160 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl\n 2026-03-20 16:26:25.650165 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs\n 2026-03-20 16:26:25.650169 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0\n 2026-03-20 16:26:25.650173 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9\n 2026-03-20 16:26:25.650184 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=\n 2026-03-20 16:26:25.650189 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.650193 | controller | CERTIFICATE-----'\n logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.650197 | controller | | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.650204 | controller | | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY\n logger.go:42: 2026-03-20 16:26:25.650208 | controller | 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx\n 2026-03-20 16:26:25.650212 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu\n 2026-03-20 16:26:25.650216 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH\n 2026-03-20 16:26:25.650220 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol\n 2026-03-20 16:26:25.650225 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X\n 2026-03-20 16:26:25.650229 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI\n 2026-03-20 16:26:25.650233 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28\n 2026-03-20 16:26:25.650237 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww\n 2026-03-20 16:26:25.650241 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/\n 2026-03-20 16:26:25.650245 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy\n 2026-03-20 16:26:25.650249 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH\n 2026-03-20 16:26:25.650254 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp\n 2026-03-20 16:26:25.650258 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6\n 2026-03-20 16:26:25.650262 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv\n 2026-03-20 16:26:25.650266 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl\n 2026-03-20 16:26:25.650274 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs\n 2026-03-20 16:26:25.650278 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0\n 2026-03-20 16:26:25.650290 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9\n 2026-03-20 16:26:25.650295 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=\n 2026-03-20 16:26:25.650299 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.650303 | controller | CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 16:19:07 2026-03-20 16:26:25.650307 | controller | | watcher-tls-certs-change/1-deploy-with-tlse | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY\n 2026-03-20 16:26:25.650312 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx\n 2026-03-20 16:26:25.650319 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu\n 2026-03-20 16:26:25.650323 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH\n 2026-03-20 16:26:25.650333 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol\n 2026-03-20 16:26:25.650338 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X\n 2026-03-20 16:26:25.650343 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI\n 2026-03-20 16:26:25.650347 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28\n 2026-03-20 16:26:25.650351 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww\n 2026-03-20 16:26:25.650356 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/\n 2026-03-20 16:26:25.650360 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy\n 2026-03-20 16:26:25.650364 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH\n 2026-03-20 16:26:25.650368 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp\n 2026-03-20 16:26:25.650372 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6\n 2026-03-20 16:26:25.650377 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv\n 2026-03-20 16:26:25.650381 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl\n 2026-03-20 16:26:25.650385 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs\n 2026-03-20 16:26:25.650389 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0\n 2026-03-20 16:26:25.650393 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9\n 2026-03-20 16:26:25.650400 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=\n 2026-03-20 16:26:25.650404 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.650408 | controller | CERTIFICATE-----' ']'\n logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.650412 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n 2026-03-20 16:26:25.650417 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert='-----BEGIN 2026-03-20 16:26:25.650421 | controller | CERTIFICATE-----\n logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.650425 | controller | | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-20 16:26:25.650429 | controller | 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx\n 2026-03-20 16:26:25.650442 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb\n 2026-03-20 16:26:25.650446 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2\n 2026-03-20 16:26:25.650450 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro\n 2026-03-20 16:26:25.650454 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671\n 2026-03-20 16:26:25.650459 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B\n 2026-03-20 16:26:25.650463 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ\n 2026-03-20 16:26:25.650467 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-20 16:26:25.650478 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E\n 2026-03-20 16:26:25.650482 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-20 16:26:25.650487 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J\n 2026-03-20 16:26:25.650491 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU\n 2026-03-20 16:26:25.650495 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq\n 2026-03-20 16:26:25.650499 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6\n 2026-03-20 16:26:25.650503 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3\n 2026-03-20 16:26:25.650508 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk\n 2026-03-20 16:26:25.650512 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb\n 2026-03-20 16:26:25.650516 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE\n 2026-03-20 16:26:25.650520 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==\n 2026-03-20 16:26:25.650524 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.650528 | controller | CERTIFICATE-----'\n logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.650533 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n 2026-03-20 16:26:25.650537 | controller | \ logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 2026-03-20 16:26:25.650541 | controller | --decode\n logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.650545 | controller | | + internal_secret_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 16:19:08 2026-03-20 16:26:25.650549 | controller | | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw\n 2026-03-20 16:26:25.650555 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx\n 2026-03-20 16:26:25.650560 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb\n 2026-03-20 16:26:25.650564 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2\n 2026-03-20 16:26:25.650571 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro\n 2026-03-20 16:26:25.650575 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671\n 2026-03-20 16:26:25.650580 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B\n 2026-03-20 16:26:25.650584 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ\n 2026-03-20 16:26:25.650588 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-20 16:26:25.650592 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E\n 2026-03-20 16:26:25.650596 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-20 16:26:25.650600 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J\n 2026-03-20 16:26:25.650605 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU\n 2026-03-20 16:26:25.650615 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq\n 2026-03-20 16:26:25.650620 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6\n 2026-03-20 16:26:25.650624 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3\n 2026-03-20 16:26:25.650629 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk\n 2026-03-20 16:26:25.650633 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb\n 2026-03-20 16:26:25.650637 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE\n 2026-03-20 16:26:25.650641 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==\n 2026-03-20 16:26:25.650645 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.650650 | controller | CERTIFICATE-----'\n logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.650654 | controller | | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.650658 | controller | | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-20 16:26:25.650662 | controller | 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx\n 2026-03-20 16:26:25.650666 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb\n 2026-03-20 16:26:25.650673 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2\n 2026-03-20 16:26:25.650677 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro\n 2026-03-20 16:26:25.650681 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671\n 2026-03-20 16:26:25.650685 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B\n 2026-03-20 16:26:25.650703 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ\n 2026-03-20 16:26:25.650710 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-20 16:26:25.650714 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E\n 2026-03-20 16:26:25.650718 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-20 16:26:25.650722 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J\n 2026-03-20 16:26:25.650727 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU\n 2026-03-20 16:26:25.650731 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq\n 2026-03-20 16:26:25.650735 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6\n 2026-03-20 16:26:25.650739 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3\n 2026-03-20 16:26:25.650743 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk\n 2026-03-20 16:26:25.650748 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb\n 2026-03-20 16:26:25.650752 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE\n 2026-03-20 16:26:25.650763 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==\n 2026-03-20 16:26:25.650769 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.650775 | controller | CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 16:19:08 2026-03-20 16:26:25.650781 | controller | | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw\n 2026-03-20 16:26:25.650786 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx\n 2026-03-20 16:26:25.650792 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb\n 2026-03-20 16:26:25.650796 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2\n 2026-03-20 16:26:25.650801 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro\n 2026-03-20 16:26:25.650807 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671\n 2026-03-20 16:26:25.650811 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B\n 2026-03-20 16:26:25.650816 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ\n 2026-03-20 16:26:25.650820 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-20 16:26:25.650827 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E\n 2026-03-20 16:26:25.650832 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-20 16:26:25.650836 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J\n 2026-03-20 16:26:25.650840 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU\n 2026-03-20 16:26:25.650844 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq\n 2026-03-20 16:26:25.650848 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6\n 2026-03-20 16:26:25.650853 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3\n 2026-03-20 16:26:25.650857 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk\n 2026-03-20 16:26:25.650861 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb\n 2026-03-20 16:26:25.650865 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE\n 2026-03-20 16:26:25.650869 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==\n 2026-03-20 16:26:25.650874 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.650878 | controller | CERTIFICATE-----' ']'\n logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse 2026-03-20 16:26:25.650882 | controller | | test step completed 1-deploy-with-tlse\n logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.650886 | controller | | starting test step 2-change-public-svc-certificate\n logger.go:42: 16:19:08 2026-03-20 16:26:25.650891 | controller | | watcher-tls-certs-change/2-change-public-svc-certificate | running command: [sh 2026-03-20 16:26:25.650895 | controller | -c oc patch Certificate -n $NAMESPACE watcher-public-svc --type='json' -p='[{\"op\": 2026-03-20 16:26:25.650899 | controller | \"replace\", \"path\": \"/spec/dnsNames\", \"value\":['watcher-public.watcher-kuttl-default.svc', 2026-03-20 16:26:25.650903 | controller | 'watcher-public.watcher-kuttl-default.svc.cluster.local']}]'\n ]\n logger.go:42: 2026-03-20 16:26:25.650914 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | certificate.cert-manager.io/watcher-public-svc 2026-03-20 16:26:25.650919 | controller | patched\n logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.650926 | controller | | running command: [sh -c set -euxo pipefail\n svc_cert=$(oc rsh -n $NAMESPACE 2026-03-20 16:26:25.650931 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n secret_cert=$(oc 2026-03-20 16:26:25.650936 | controller | get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-20 16:26:25.650942 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-20 16:26:25.650948 | controller | \ if [ \"${svc_cert}\" != \"${secret_cert}\" ]; then\n exit 1\n fi\n 2026-03-20 16:26:25.650954 | controller | \ ]\n logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.650959 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-20 16:26:25.650965 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.650971 | controller | | + svc_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.650976 | controller | | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY\n logger.go:42: 2026-03-20 16:26:25.650980 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx\n 2026-03-20 16:26:25.650984 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.650988 | controller | | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu\n logger.go:42: 2026-03-20 16:26:25.650993 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH\n 2026-03-20 16:26:25.650997 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651001 | controller | | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol\n logger.go:42: 2026-03-20 16:26:25.651007 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X\n 2026-03-20 16:26:25.651012 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651017 | controller | | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI\n logger.go:42: 2026-03-20 16:26:25.651021 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28\n 2026-03-20 16:26:25.651027 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651031 | controller | | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww\n logger.go:42: 2026-03-20 16:26:25.651035 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/\n 2026-03-20 16:26:25.651040 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651044 | controller | | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy\n logger.go:42: 2026-03-20 16:26:25.651048 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH\n 2026-03-20 16:26:25.651052 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651056 | controller | | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp\n logger.go:42: 2026-03-20 16:26:25.651060 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6\n 2026-03-20 16:26:25.651065 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651069 | controller | | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv\n logger.go:42: 2026-03-20 16:26:25.651076 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl\n 2026-03-20 16:26:25.651080 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651084 | controller | | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs\n logger.go:42: 2026-03-20 16:26:25.651088 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0\n 2026-03-20 16:26:25.651093 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651097 | controller | | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9\n logger.go:42: 2026-03-20 16:26:25.651120 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=\n 2026-03-20 16:26:25.651126 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651132 | controller | | -----END CERTIFICATE-----'\n logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651137 | controller | | ++ base64 --decode\n logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651142 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n 2026-03-20 16:26:25.651146 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651150 | controller | | + secret_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651154 | controller | | MIID6jCCAlKgAwIBAgIQZJaVw8Wo/ookNMvl5tSf7zANBgkqhkiG9w0BAQwFADAY\n logger.go:42: 2026-03-20 16:26:25.651159 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTkwOFoXDTMxMDMx\n 2026-03-20 16:26:25.651163 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651167 | controller | | OTE2MTkwOFowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKXRB7B3\n logger.go:42: 2026-03-20 16:26:25.651179 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | rhfYNMtE5MNaVBPTQ1ChoGf12bhDR7Hfz4Pv0e9XQRR0iY1L3O/sIYBsUs+UDZIi\n 2026-03-20 16:26:25.651184 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651188 | controller | | 1sO1e0hu0f/ub72rJMdYIWEW+gWSTmTOFffkt+dnobBgQlXXw5g7ZSg4EbaGx6Kf\n logger.go:42: 2026-03-20 16:26:25.651192 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | GKGVlEhQ9uEjQNrifD33D8vfR/3AsqR1MnZEcrQRXHjjhDXlhLNtNQEceenSDm+0\n 2026-03-20 16:26:25.651196 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651200 | controller | | 0ok9qcjT0JDZCVre77nyPUISheJeWK3ZkiHxDthkL7ukO/9rVwYRNuWEB4JHZHFI\n logger.go:42: 2026-03-20 16:26:25.651205 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | MqGuhbe7jgVW0QrBl314Zjw5ET2FKdlTKtd/b+PhNnuU2qhjWYO76WhqcpllFwau\n 2026-03-20 16:26:25.651209 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651213 | controller | | sunqw0flNMujB88CAwEAAaOBxzCBxDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww\n logger.go:42: 2026-03-20 16:26:25.651217 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/\n 2026-03-20 16:26:25.651222 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651226 | controller | | 0Hm6PaQEE6ByeDBuBgNVHREBAf8EZDBigih3YXRjaGVyLXB1YmxpYy53YXRjaGVy\n logger.go:42: 2026-03-20 16:26:25.651233 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | LWt1dHRsLWRlZmF1bHQuc3ZjgjZ3YXRjaGVyLXB1YmxpYy53YXRjaGVyLWt1dHRs\n 2026-03-20 16:26:25.651245 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651250 | controller | | LWRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwwDQYJKoZIhvcNAQEMBQADggGBAC/o\n logger.go:42: 2026-03-20 16:26:25.651254 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | 5/i5C0Y/AKLiB3xtslI8c9UVyiOwIpEOrBpAYh0CQe7PkZ9zV4mBLHrY5tneZeQ/\n 2026-03-20 16:26:25.651258 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651262 | controller | | gKY2iSGZifksOo9R2RmPjpKPIrDqgKveIqRYsNeCkVKKXimGwzNVBhoCDnpkrd0r\n logger.go:42: 2026-03-20 16:26:25.651266 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | HttXDFUG47HTtNbtx9gbHEnL9JViBLG/91Fb2PavY1G6rYeRpJT13dAr2/ozwk0F\n 2026-03-20 16:26:25.651271 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651275 | controller | | j5lPZ2ZxWmWxE51A3yAmXXa8NT+MJpwWW1Gu9eSb+Bn2TELEMuEHos6Qn6Ob957P\n logger.go:42: 2026-03-20 16:26:25.651279 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | YcG0oFRp+ecTVjfLhZmXMyYoEzmLlXLuGaUQqvlXECGTSbw5erNtKRkS4gAQ0PAI\n 2026-03-20 16:26:25.651286 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651290 | controller | | sRSypaVKNLM96+09mRoN9ppbVXUrmd4haOg9l09Bpth6s6bywa3GcqkBHjORfRUu\n logger.go:42: 2026-03-20 16:26:25.651295 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | 6/lzDFbw+LAZRB7kiWeyD9W+LlhYC6ZzxjrTpuNIfNwmsCKCPzHadwq2OXa2vIZi\n 2026-03-20 16:26:25.651307 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651317 | controller | | p+EfRtIGSWnMfKbcf6w2QsDS8061IT7HHoFOS9K8BWVPA3gUgAn4f0+B5r5t1g==\n logger.go:42: 2026-03-20 16:26:25.651322 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | -----END CERTIFICATE-----'\n 2026-03-20 16:26:25.651326 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651331 | controller | | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651335 | controller | | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY\n logger.go:42: 2026-03-20 16:26:25.651340 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx\n 2026-03-20 16:26:25.651344 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651348 | controller | | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu\n logger.go:42: 2026-03-20 16:26:25.651352 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH\n 2026-03-20 16:26:25.651356 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651360 | controller | | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol\n logger.go:42: 2026-03-20 16:26:25.651365 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X\n 2026-03-20 16:26:25.651369 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651373 | controller | | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI\n logger.go:42: 2026-03-20 16:26:25.651379 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28\n 2026-03-20 16:26:25.651383 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651390 | controller | | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww\n logger.go:42: 2026-03-20 16:26:25.651394 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/\n 2026-03-20 16:26:25.651398 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651403 | controller | | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy\n logger.go:42: 2026-03-20 16:26:25.651407 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH\n 2026-03-20 16:26:25.651411 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651415 | controller | | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp\n logger.go:42: 2026-03-20 16:26:25.651419 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6\n 2026-03-20 16:26:25.651424 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651428 | controller | | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv\n logger.go:42: 2026-03-20 16:26:25.651432 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl\n 2026-03-20 16:26:25.651436 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651441 | controller | | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs\n logger.go:42: 2026-03-20 16:26:25.651446 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0\n 2026-03-20 16:26:25.651452 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651457 | controller | | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9\n logger.go:42: 2026-03-20 16:26:25.651463 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=\n 2026-03-20 16:26:25.651467 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651472 | controller | | -----END CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 2026-03-20 16:26:25.651476 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | MIID6jCCAlKgAwIBAgIQZJaVw8Wo/ookNMvl5tSf7zANBgkqhkiG9w0BAQwFADAY\n 2026-03-20 16:26:25.651480 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651484 | controller | | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTkwOFoXDTMxMDMx\n logger.go:42: 2026-03-20 16:26:25.651495 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | OTE2MTkwOFowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKXRB7B3\n 2026-03-20 16:26:25.651500 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651504 | controller | | rhfYNMtE5MNaVBPTQ1ChoGf12bhDR7Hfz4Pv0e9XQRR0iY1L3O/sIYBsUs+UDZIi\n logger.go:42: 2026-03-20 16:26:25.651508 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | 1sO1e0hu0f/ub72rJMdYIWEW+gWSTmTOFffkt+dnobBgQlXXw5g7ZSg4EbaGx6Kf\n 2026-03-20 16:26:25.651513 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651519 | controller | | GKGVlEhQ9uEjQNrifD33D8vfR/3AsqR1MnZEcrQRXHjjhDXlhLNtNQEceenSDm+0\n logger.go:42: 2026-03-20 16:26:25.651523 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | 0ok9qcjT0JDZCVre77nyPUISheJeWK3ZkiHxDthkL7ukO/9rVwYRNuWEB4JHZHFI\n 2026-03-20 16:26:25.651528 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651535 | controller | | MqGuhbe7jgVW0QrBl314Zjw5ET2FKdlTKtd/b+PhNnuU2qhjWYO76WhqcpllFwau\n logger.go:42: 2026-03-20 16:26:25.651539 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | sunqw0flNMujB88CAwEAAaOBxzCBxDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww\n 2026-03-20 16:26:25.651749 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651758 | controller | | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/\n logger.go:42: 2026-03-20 16:26:25.651764 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | 0Hm6PaQEE6ByeDBuBgNVHREBAf8EZDBigih3YXRjaGVyLXB1YmxpYy53YXRjaGVy\n 2026-03-20 16:26:25.651770 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651775 | controller | | LWt1dHRsLWRlZmF1bHQuc3ZjgjZ3YXRjaGVyLXB1YmxpYy53YXRjaGVyLWt1dHRs\n logger.go:42: 2026-03-20 16:26:25.651781 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | LWRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwwDQYJKoZIhvcNAQEMBQADggGBAC/o\n 2026-03-20 16:26:25.651787 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651792 | controller | | 5/i5C0Y/AKLiB3xtslI8c9UVyiOwIpEOrBpAYh0CQe7PkZ9zV4mBLHrY5tneZeQ/\n logger.go:42: 2026-03-20 16:26:25.651798 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | gKY2iSGZifksOo9R2RmPjpKPIrDqgKveIqRYsNeCkVKKXimGwzNVBhoCDnpkrd0r\n 2026-03-20 16:26:25.651802 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651806 | controller | | HttXDFUG47HTtNbtx9gbHEnL9JViBLG/91Fb2PavY1G6rYeRpJT13dAr2/ozwk0F\n logger.go:42: 2026-03-20 16:26:25.651810 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | j5lPZ2ZxWmWxE51A3yAmXXa8NT+MJpwWW1Gu9eSb+Bn2TELEMuEHos6Qn6Ob957P\n 2026-03-20 16:26:25.651814 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651819 | controller | | YcG0oFRp+ecTVjfLhZmXMyYoEzmLlXLuGaUQqvlXECGTSbw5erNtKRkS4gAQ0PAI\n logger.go:42: 2026-03-20 16:26:25.651823 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | sRSypaVKNLM96+09mRoN9ppbVXUrmd4haOg9l09Bpth6s6bywa3GcqkBHjORfRUu\n 2026-03-20 16:26:25.651827 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651831 | controller | | 6/lzDFbw+LAZRB7kiWeyD9W+LlhYC6ZzxjrTpuNIfNwmsCKCPzHadwq2OXa2vIZi\n logger.go:42: 2026-03-20 16:26:25.651835 | controller | 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate | p+EfRtIGSWnMfKbcf6w2QsDS8061IT7HHoFOS9K8BWVPA3gUgAn4f0+B5r5t1g==\n 2026-03-20 16:26:25.651839 | controller | \ logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651843 | controller | | -----END CERTIFICATE-----' ']'\n logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651847 | controller | | + exit 1\n logger.go:42: 16:19:09 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651852 | controller | | running command: [sh -c set -euxo pipefail\n svc_cert=$(oc rsh -n $NAMESPACE 2026-03-20 16:26:25.651864 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n secret_cert=$(oc 2026-03-20 16:26:25.651868 | controller | get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-20 16:26:25.651873 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-20 16:26:25.651877 | controller | \ if [ \"${svc_cert}\" != \"${secret_cert}\" ]; then\n exit 1\n fi\n 2026-03-20 16:26:25.651881 | controller | \ ]\n logger.go:42: 16:19:09 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651885 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-20 16:26:25.651903 | controller | \ logger.go:42: 16:19:09 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651907 | controller | | error: unable to upgrade connection: container not found (\"watcher-api\")\n logger.go:42: 2026-03-20 16:26:25.651912 | controller | 16:19:09 | watcher-tls-certs-change/2-change-public-svc-certificate | + svc_cert=\n 2026-03-20 16:26:25.651917 | controller | \ logger.go:42: 16:19:10 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651921 | controller | | running command: [sh -c set -euxo pipefail\n svc_cert=$(oc rsh -n $NAMESPACE 2026-03-20 16:26:25.651926 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n secret_cert=$(oc 2026-03-20 16:26:25.651931 | controller | get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-20 16:26:25.651935 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-20 16:26:25.651939 | controller | \ if [ \"${svc_cert}\" != \"${secret_cert}\" ]; then\n exit 1\n fi\n 2026-03-20 16:26:25.651946 | controller | \ ]\n logger.go:42: 16:19:10 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651950 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-20 16:26:25.651954 | controller | \ logger.go:42: 16:19:10 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651959 | controller | | error: unable to upgrade connection: container not found (\"watcher-api\")\n logger.go:42: 2026-03-20 16:26:25.651963 | controller | 16:19:10 | watcher-tls-certs-change/2-change-public-svc-certificate | + svc_cert=\n 2026-03-20 16:26:25.651967 | controller | \ logger.go:42: 16:19:11 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651971 | controller | | running command: [sh -c set -euxo pipefail\n svc_cert=$(oc rsh -n $NAMESPACE 2026-03-20 16:26:25.651975 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt)\n secret_cert=$(oc 2026-03-20 16:26:25.651979 | controller | get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-20 16:26:25.651984 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-20 16:26:25.651988 | controller | \ if [ \"${svc_cert}\" != \"${secret_cert}\" ]; then\n exit 1\n fi\n 2026-03-20 16:26:25.651992 | controller | \ ]\n logger.go:42: 16:19:11 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.651996 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt\n 2026-03-20 16:26:25.652002 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652007 | controller | | + svc_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652019 | controller | | MIID6jCCAlKgAwIBAgIQZJaVw8Wo/ookNMvl5tSf7zANBgkqhkiG9w0BAQwFADAY\n logger.go:42: 2026-03-20 16:26:25.652024 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTkwOFoXDTMxMDMx\n 2026-03-20 16:26:25.652028 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652033 | controller | | OTE2MTkwOFowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKXRB7B3\n logger.go:42: 2026-03-20 16:26:25.652037 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | rhfYNMtE5MNaVBPTQ1ChoGf12bhDR7Hfz4Pv0e9XQRR0iY1L3O/sIYBsUs+UDZIi\n 2026-03-20 16:26:25.652041 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652045 | controller | | 1sO1e0hu0f/ub72rJMdYIWEW+gWSTmTOFffkt+dnobBgQlXXw5g7ZSg4EbaGx6Kf\n logger.go:42: 2026-03-20 16:26:25.652049 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | GKGVlEhQ9uEjQNrifD33D8vfR/3AsqR1MnZEcrQRXHjjhDXlhLNtNQEceenSDm+0\n 2026-03-20 16:26:25.652056 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652060 | controller | | 0ok9qcjT0JDZCVre77nyPUISheJeWK3ZkiHxDthkL7ukO/9rVwYRNuWEB4JHZHFI\n logger.go:42: 2026-03-20 16:26:25.652064 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | MqGuhbe7jgVW0QrBl314Zjw5ET2FKdlTKtd/b+PhNnuU2qhjWYO76WhqcpllFwau\n 2026-03-20 16:26:25.652068 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652072 | controller | | sunqw0flNMujB88CAwEAAaOBxzCBxDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww\n logger.go:42: 2026-03-20 16:26:25.652077 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/\n 2026-03-20 16:26:25.652081 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652087 | controller | | 0Hm6PaQEE6ByeDBuBgNVHREBAf8EZDBigih3YXRjaGVyLXB1YmxpYy53YXRjaGVy\n logger.go:42: 2026-03-20 16:26:25.652100 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | LWt1dHRsLWRlZmF1bHQuc3ZjgjZ3YXRjaGVyLXB1YmxpYy53YXRjaGVyLWt1dHRs\n 2026-03-20 16:26:25.652104 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652108 | controller | | LWRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwwDQYJKoZIhvcNAQEMBQADggGBAC/o\n logger.go:42: 2026-03-20 16:26:25.652113 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | 5/i5C0Y/AKLiB3xtslI8c9UVyiOwIpEOrBpAYh0CQe7PkZ9zV4mBLHrY5tneZeQ/\n 2026-03-20 16:26:25.652117 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652121 | controller | | gKY2iSGZifksOo9R2RmPjpKPIrDqgKveIqRYsNeCkVKKXimGwzNVBhoCDnpkrd0r\n logger.go:42: 2026-03-20 16:26:25.652125 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | HttXDFUG47HTtNbtx9gbHEnL9JViBLG/91Fb2PavY1G6rYeRpJT13dAr2/ozwk0F\n 2026-03-20 16:26:25.652130 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652134 | controller | | j5lPZ2ZxWmWxE51A3yAmXXa8NT+MJpwWW1Gu9eSb+Bn2TELEMuEHos6Qn6Ob957P\n logger.go:42: 2026-03-20 16:26:25.652138 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | YcG0oFRp+ecTVjfLhZmXMyYoEzmLlXLuGaUQqvlXECGTSbw5erNtKRkS4gAQ0PAI\n 2026-03-20 16:26:25.652142 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652146 | controller | | sRSypaVKNLM96+09mRoN9ppbVXUrmd4haOg9l09Bpth6s6bywa3GcqkBHjORfRUu\n logger.go:42: 2026-03-20 16:26:25.652150 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | 6/lzDFbw+LAZRB7kiWeyD9W+LlhYC6ZzxjrTpuNIfNwmsCKCPzHadwq2OXa2vIZi\n 2026-03-20 16:26:25.652155 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652159 | controller | | p+EfRtIGSWnMfKbcf6w2QsDS8061IT7HHoFOS9K8BWVPA3gUgAn4f0+B5r5t1g==\n logger.go:42: 2026-03-20 16:26:25.652163 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | -----END CERTIFICATE-----'\n 2026-03-20 16:26:25.652167 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652171 | controller | | ++ base64 --decode\n logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652176 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o 'jsonpath={.data.tls\\.crt}'\n 2026-03-20 16:26:25.652180 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652184 | controller | | + secret_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652191 | controller | | MIID6jCCAlKgAwIBAgIQZJaVw8Wo/ookNMvl5tSf7zANBgkqhkiG9w0BAQwFADAY\n logger.go:42: 2026-03-20 16:26:25.652195 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTkwOFoXDTMxMDMx\n 2026-03-20 16:26:25.652199 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652204 | controller | | OTE2MTkwOFowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKXRB7B3\n logger.go:42: 2026-03-20 16:26:25.652208 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | rhfYNMtE5MNaVBPTQ1ChoGf12bhDR7Hfz4Pv0e9XQRR0iY1L3O/sIYBsUs+UDZIi\n 2026-03-20 16:26:25.652212 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652216 | controller | | 1sO1e0hu0f/ub72rJMdYIWEW+gWSTmTOFffkt+dnobBgQlXXw5g7ZSg4EbaGx6Kf\n logger.go:42: 2026-03-20 16:26:25.652221 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | GKGVlEhQ9uEjQNrifD33D8vfR/3AsqR1MnZEcrQRXHjjhDXlhLNtNQEceenSDm+0\n 2026-03-20 16:26:25.652225 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652229 | controller | | 0ok9qcjT0JDZCVre77nyPUISheJeWK3ZkiHxDthkL7ukO/9rVwYRNuWEB4JHZHFI\n logger.go:42: 2026-03-20 16:26:25.652233 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | MqGuhbe7jgVW0QrBl314Zjw5ET2FKdlTKtd/b+PhNnuU2qhjWYO76WhqcpllFwau\n 2026-03-20 16:26:25.652239 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652243 | controller | | sunqw0flNMujB88CAwEAAaOBxzCBxDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww\n logger.go:42: 2026-03-20 16:26:25.652248 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/\n 2026-03-20 16:26:25.652252 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652256 | controller | | 0Hm6PaQEE6ByeDBuBgNVHREBAf8EZDBigih3YXRjaGVyLXB1YmxpYy53YXRjaGVy\n logger.go:42: 2026-03-20 16:26:25.652260 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | LWt1dHRsLWRlZmF1bHQuc3ZjgjZ3YXRjaGVyLXB1YmxpYy53YXRjaGVyLWt1dHRs\n 2026-03-20 16:26:25.652271 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652276 | controller | | LWRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwwDQYJKoZIhvcNAQEMBQADggGBAC/o\n logger.go:42: 2026-03-20 16:26:25.652280 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | 5/i5C0Y/AKLiB3xtslI8c9UVyiOwIpEOrBpAYh0CQe7PkZ9zV4mBLHrY5tneZeQ/\n 2026-03-20 16:26:25.652284 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652288 | controller | | gKY2iSGZifksOo9R2RmPjpKPIrDqgKveIqRYsNeCkVKKXimGwzNVBhoCDnpkrd0r\n logger.go:42: 2026-03-20 16:26:25.652293 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | HttXDFUG47HTtNbtx9gbHEnL9JViBLG/91Fb2PavY1G6rYeRpJT13dAr2/ozwk0F\n 2026-03-20 16:26:25.652297 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652301 | controller | | j5lPZ2ZxWmWxE51A3yAmXXa8NT+MJpwWW1Gu9eSb+Bn2TELEMuEHos6Qn6Ob957P\n logger.go:42: 2026-03-20 16:26:25.652305 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | YcG0oFRp+ecTVjfLhZmXMyYoEzmLlXLuGaUQqvlXECGTSbw5erNtKRkS4gAQ0PAI\n 2026-03-20 16:26:25.652309 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652313 | controller | | sRSypaVKNLM96+09mRoN9ppbVXUrmd4haOg9l09Bpth6s6bywa3GcqkBHjORfRUu\n logger.go:42: 2026-03-20 16:26:25.652318 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | 6/lzDFbw+LAZRB7kiWeyD9W+LlhYC6ZzxjrTpuNIfNwmsCKCPzHadwq2OXa2vIZi\n 2026-03-20 16:26:25.652322 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652329 | controller | | p+EfRtIGSWnMfKbcf6w2QsDS8061IT7HHoFOS9K8BWVPA3gUgAn4f0+B5r5t1g==\n logger.go:42: 2026-03-20 16:26:25.652333 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | -----END CERTIFICATE-----'\n 2026-03-20 16:26:25.652337 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652341 | controller | | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652346 | controller | | MIID6jCCAlKgAwIBAgIQZJaVw8Wo/ookNMvl5tSf7zANBgkqhkiG9w0BAQwFADAY\n logger.go:42: 2026-03-20 16:26:25.652350 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTkwOFoXDTMxMDMx\n 2026-03-20 16:26:25.652354 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652358 | controller | | OTE2MTkwOFowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKXRB7B3\n logger.go:42: 2026-03-20 16:26:25.652362 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | rhfYNMtE5MNaVBPTQ1ChoGf12bhDR7Hfz4Pv0e9XQRR0iY1L3O/sIYBsUs+UDZIi\n 2026-03-20 16:26:25.652367 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652371 | controller | | 1sO1e0hu0f/ub72rJMdYIWEW+gWSTmTOFffkt+dnobBgQlXXw5g7ZSg4EbaGx6Kf\n logger.go:42: 2026-03-20 16:26:25.652375 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | GKGVlEhQ9uEjQNrifD33D8vfR/3AsqR1MnZEcrQRXHjjhDXlhLNtNQEceenSDm+0\n 2026-03-20 16:26:25.652379 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652383 | controller | | 0ok9qcjT0JDZCVre77nyPUISheJeWK3ZkiHxDthkL7ukO/9rVwYRNuWEB4JHZHFI\n logger.go:42: 2026-03-20 16:26:25.652387 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | MqGuhbe7jgVW0QrBl314Zjw5ET2FKdlTKtd/b+PhNnuU2qhjWYO76WhqcpllFwau\n 2026-03-20 16:26:25.652392 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652396 | controller | | sunqw0flNMujB88CAwEAAaOBxzCBxDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww\n logger.go:42: 2026-03-20 16:26:25.652400 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/\n 2026-03-20 16:26:25.652404 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652408 | controller | | 0Hm6PaQEE6ByeDBuBgNVHREBAf8EZDBigih3YXRjaGVyLXB1YmxpYy53YXRjaGVy\n logger.go:42: 2026-03-20 16:26:25.652416 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | LWt1dHRsLWRlZmF1bHQuc3ZjgjZ3YXRjaGVyLXB1YmxpYy53YXRjaGVyLWt1dHRs\n 2026-03-20 16:26:25.652420 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652424 | controller | | LWRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwwDQYJKoZIhvcNAQEMBQADggGBAC/o\n logger.go:42: 2026-03-20 16:26:25.652429 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | 5/i5C0Y/AKLiB3xtslI8c9UVyiOwIpEOrBpAYh0CQe7PkZ9zV4mBLHrY5tneZeQ/\n 2026-03-20 16:26:25.652433 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652444 | controller | | gKY2iSGZifksOo9R2RmPjpKPIrDqgKveIqRYsNeCkVKKXimGwzNVBhoCDnpkrd0r\n logger.go:42: 2026-03-20 16:26:25.652450 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | HttXDFUG47HTtNbtx9gbHEnL9JViBLG/91Fb2PavY1G6rYeRpJT13dAr2/ozwk0F\n 2026-03-20 16:26:25.652454 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652463 | controller | | j5lPZ2ZxWmWxE51A3yAmXXa8NT+MJpwWW1Gu9eSb+Bn2TELEMuEHos6Qn6Ob957P\n logger.go:42: 2026-03-20 16:26:25.652468 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | YcG0oFRp+ecTVjfLhZmXMyYoEzmLlXLuGaUQqvlXECGTSbw5erNtKRkS4gAQ0PAI\n 2026-03-20 16:26:25.652475 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652479 | controller | | sRSypaVKNLM96+09mRoN9ppbVXUrmd4haOg9l09Bpth6s6bywa3GcqkBHjORfRUu\n logger.go:42: 2026-03-20 16:26:25.652483 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | 6/lzDFbw+LAZRB7kiWeyD9W+LlhYC6ZzxjrTpuNIfNwmsCKCPzHadwq2OXa2vIZi\n 2026-03-20 16:26:25.652488 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652492 | controller | | p+EfRtIGSWnMfKbcf6w2QsDS8061IT7HHoFOS9K8BWVPA3gUgAn4f0+B5r5t1g==\n logger.go:42: 2026-03-20 16:26:25.652496 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | -----END CERTIFICATE-----' 2026-03-20 16:26:25.652500 | controller | '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652504 | controller | | MIID6jCCAlKgAwIBAgIQZJaVw8Wo/ookNMvl5tSf7zANBgkqhkiG9w0BAQwFADAY\n logger.go:42: 2026-03-20 16:26:25.652509 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTkwOFoXDTMxMDMx\n 2026-03-20 16:26:25.652513 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652517 | controller | | OTE2MTkwOFowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKXRB7B3\n logger.go:42: 2026-03-20 16:26:25.652521 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | rhfYNMtE5MNaVBPTQ1ChoGf12bhDR7Hfz4Pv0e9XQRR0iY1L3O/sIYBsUs+UDZIi\n 2026-03-20 16:26:25.652526 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652530 | controller | | 1sO1e0hu0f/ub72rJMdYIWEW+gWSTmTOFffkt+dnobBgQlXXw5g7ZSg4EbaGx6Kf\n logger.go:42: 2026-03-20 16:26:25.652534 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | GKGVlEhQ9uEjQNrifD33D8vfR/3AsqR1MnZEcrQRXHjjhDXlhLNtNQEceenSDm+0\n 2026-03-20 16:26:25.652538 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652542 | controller | | 0ok9qcjT0JDZCVre77nyPUISheJeWK3ZkiHxDthkL7ukO/9rVwYRNuWEB4JHZHFI\n logger.go:42: 2026-03-20 16:26:25.652547 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | MqGuhbe7jgVW0QrBl314Zjw5ET2FKdlTKtd/b+PhNnuU2qhjWYO76WhqcpllFwau\n 2026-03-20 16:26:25.652551 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652555 | controller | | sunqw0flNMujB88CAwEAAaOBxzCBxDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww\n logger.go:42: 2026-03-20 16:26:25.652559 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/\n 2026-03-20 16:26:25.652563 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652567 | controller | | 0Hm6PaQEE6ByeDBuBgNVHREBAf8EZDBigih3YXRjaGVyLXB1YmxpYy53YXRjaGVy\n logger.go:42: 2026-03-20 16:26:25.652572 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | LWt1dHRsLWRlZmF1bHQuc3ZjgjZ3YXRjaGVyLXB1YmxpYy53YXRjaGVyLWt1dHRs\n 2026-03-20 16:26:25.652576 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652580 | controller | | LWRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwwDQYJKoZIhvcNAQEMBQADggGBAC/o\n logger.go:42: 2026-03-20 16:26:25.652584 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | 5/i5C0Y/AKLiB3xtslI8c9UVyiOwIpEOrBpAYh0CQe7PkZ9zV4mBLHrY5tneZeQ/\n 2026-03-20 16:26:25.652588 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652592 | controller | | gKY2iSGZifksOo9R2RmPjpKPIrDqgKveIqRYsNeCkVKKXimGwzNVBhoCDnpkrd0r\n logger.go:42: 2026-03-20 16:26:25.652597 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | HttXDFUG47HTtNbtx9gbHEnL9JViBLG/91Fb2PavY1G6rYeRpJT13dAr2/ozwk0F\n 2026-03-20 16:26:25.652602 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652607 | controller | | j5lPZ2ZxWmWxE51A3yAmXXa8NT+MJpwWW1Gu9eSb+Bn2TELEMuEHos6Qn6Ob957P\n logger.go:42: 2026-03-20 16:26:25.652619 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | YcG0oFRp+ecTVjfLhZmXMyYoEzmLlXLuGaUQqvlXECGTSbw5erNtKRkS4gAQ0PAI\n 2026-03-20 16:26:25.652623 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652627 | controller | | sRSypaVKNLM96+09mRoN9ppbVXUrmd4haOg9l09Bpth6s6bywa3GcqkBHjORfRUu\n logger.go:42: 2026-03-20 16:26:25.652631 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | 6/lzDFbw+LAZRB7kiWeyD9W+LlhYC6ZzxjrTpuNIfNwmsCKCPzHadwq2OXa2vIZi\n 2026-03-20 16:26:25.652636 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652640 | controller | | p+EfRtIGSWnMfKbcf6w2QsDS8061IT7HHoFOS9K8BWVPA3gUgAn4f0+B5r5t1g==\n logger.go:42: 2026-03-20 16:26:25.652644 | controller | 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate | -----END CERTIFICATE-----' 2026-03-20 16:26:25.652648 | controller | ']'\n logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.652653 | controller | | test step completed 2-change-public-svc-certificate\n logger.go:42: 16:19:12 2026-03-20 16:26:25.652657 | controller | | watcher-tls-certs-change/3-change-internal-svc-certificate | starting test step 2026-03-20 16:26:25.652661 | controller | 3-change-internal-svc-certificate\n logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.652665 | controller | | running command: [sh -c oc patch Certificate -n $NAMESPACE watcher-internal-svc 2026-03-20 16:26:25.652669 | controller | --type='json' -p='[{\"op\": \"replace\", \"path\": \"/spec/dnsNames\", \"value\":['watcher-internal.watcher-kuttl-default.svc', 2026-03-20 16:26:25.652673 | controller | 'watcher-internal.watcher-kuttl-default.svc.cluster.local']}]'\n ]\n logger.go:42: 2026-03-20 16:26:25.652678 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | certificate.cert-manager.io/watcher-internal-svc 2026-03-20 16:26:25.652682 | controller | patched\n logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.652686 | controller | | running command: [sh -c set -euxo pipefail\n svc_cert=$(oc rsh -n $NAMESPACE 2026-03-20 16:26:25.652690 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt)\n secret_cert=$(oc 2026-03-20 16:26:25.652706 | controller | get -n $NAMESPACE secret cert-watcher-internal-svc -o jsonpath='{.data.tls\\.crt}' 2026-03-20 16:26:25.652713 | controller | | base64 --decode)\n # ensure that the svc secret and cert secret match\n 2026-03-20 16:26:25.652717 | controller | \ if [ \"${svc_cert}\" != \"${secret_cert}\" ]; then\n exit 1\n fi\n 2026-03-20 16:26:25.652722 | controller | \ ]\n logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.652726 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt\n 2026-03-20 16:26:25.652730 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.652734 | controller | | + svc_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.652738 | controller | | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-20 16:26:25.652743 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx\n 2026-03-20 16:26:25.652747 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.652753 | controller | | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb\n logger.go:42: 2026-03-20 16:26:25.652757 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2\n 2026-03-20 16:26:25.652761 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.652765 | controller | | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro\n logger.go:42: 2026-03-20 16:26:25.652770 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671\n 2026-03-20 16:26:25.652774 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.652778 | controller | | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B\n logger.go:42: 2026-03-20 16:26:25.652782 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ\n 2026-03-20 16:26:25.652786 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.652791 | controller | | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n logger.go:42: 2026-03-20 16:26:25.652795 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E\n 2026-03-20 16:26:25.652807 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.652811 | controller | | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n logger.go:42: 2026-03-20 16:26:25.652816 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J\n 2026-03-20 16:26:25.652820 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.652824 | controller | | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU\n logger.go:42: 2026-03-20 16:26:25.652828 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq\n 2026-03-20 16:26:25.652833 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.652837 | controller | | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6\n logger.go:42: 2026-03-20 16:26:25.652841 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3\n 2026-03-20 16:26:25.652845 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.652849 | controller | | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk\n logger.go:42: 2026-03-20 16:26:25.652854 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb\n 2026-03-20 16:26:25.652858 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.652862 | controller | | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE\n logger.go:42: 2026-03-20 16:26:25.652866 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==\n 2026-03-20 16:26:25.652870 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.652875 | controller | | -----END CERTIFICATE-----'\n logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.652879 | controller | | ++ base64 --decode\n logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.652885 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o 'jsonpath={.data.tls\\.crt}'\n 2026-03-20 16:26:25.652889 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.652893 | controller | | + secret_cert='-----BEGIN CERTIFICATE-----\n logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.652898 | controller | | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-20 16:26:25.652902 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx\n 2026-03-20 16:26:25.652909 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.652913 | controller | | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb\n logger.go:42: 2026-03-20 16:26:25.652917 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2\n 2026-03-20 16:26:25.652922 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.652926 | controller | | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro\n logger.go:42: 2026-03-20 16:26:25.652930 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671\n 2026-03-20 16:26:25.652934 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.652939 | controller | | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B\n logger.go:42: 2026-03-20 16:26:25.652943 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ\n 2026-03-20 16:26:25.652947 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.652953 | controller | | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n logger.go:42: 2026-03-20 16:26:25.652957 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E\n 2026-03-20 16:26:25.652961 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.652966 | controller | | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n logger.go:42: 2026-03-20 16:26:25.652976 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J\n 2026-03-20 16:26:25.652981 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.652986 | controller | | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU\n logger.go:42: 2026-03-20 16:26:25.652990 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq\n 2026-03-20 16:26:25.652995 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.652999 | controller | | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6\n logger.go:42: 2026-03-20 16:26:25.653003 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3\n 2026-03-20 16:26:25.653007 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.653011 | controller | | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk\n logger.go:42: 2026-03-20 16:26:25.653016 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb\n 2026-03-20 16:26:25.653020 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.653025 | controller | | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE\n logger.go:42: 2026-03-20 16:26:25.653030 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==\n 2026-03-20 16:26:25.653034 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.653038 | controller | | -----END CERTIFICATE-----'\n logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.653042 | controller | | + '[' '-----BEGIN CERTIFICATE-----\n logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.653047 | controller | | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw\n logger.go:42: 2026-03-20 16:26:25.653051 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx\n 2026-03-20 16:26:25.653055 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.653059 | controller | | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb\n logger.go:42: 2026-03-20 16:26:25.653064 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2\n 2026-03-20 16:26:25.653068 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.653072 | controller | | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro\n logger.go:42: 2026-03-20 16:26:25.653076 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671\n 2026-03-20 16:26:25.653080 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.653084 | controller | | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B\n logger.go:42: 2026-03-20 16:26:25.653089 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ\n 2026-03-20 16:26:25.653093 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.653097 | controller | | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n logger.go:42: 2026-03-20 16:26:25.653101 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E\n 2026-03-20 16:26:25.653105 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.653110 | controller | | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n logger.go:42: 2026-03-20 16:26:25.653114 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J\n 2026-03-20 16:26:25.653118 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.653122 | controller | | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU\n logger.go:42: 2026-03-20 16:26:25.653126 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq\n 2026-03-20 16:26:25.653137 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.653142 | controller | | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6\n logger.go:42: 2026-03-20 16:26:25.653147 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3\n 2026-03-20 16:26:25.653151 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.653157 | controller | | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk\n logger.go:42: 2026-03-20 16:26:25.653161 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb\n 2026-03-20 16:26:25.653166 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.653170 | controller | | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE\n logger.go:42: 2026-03-20 16:26:25.653174 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==\n 2026-03-20 16:26:25.653188 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.653192 | controller | | -----END CERTIFICATE-----' '!=' '-----BEGIN CERTIFICATE-----\n logger.go:42: 2026-03-20 16:26:25.653197 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw\n 2026-03-20 16:26:25.653201 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.653205 | controller | | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx\n logger.go:42: 2026-03-20 16:26:25.653209 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb\n 2026-03-20 16:26:25.653213 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.653217 | controller | | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2\n logger.go:42: 2026-03-20 16:26:25.653222 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro\n 2026-03-20 16:26:25.653226 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.653230 | controller | | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671\n logger.go:42: 2026-03-20 16:26:25.653234 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B\n 2026-03-20 16:26:25.653240 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.653245 | controller | | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ\n logger.go:42: 2026-03-20 16:26:25.653249 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l\n 2026-03-20 16:26:25.653253 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.653257 | controller | | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E\n logger.go:42: 2026-03-20 16:26:25.653261 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh\n 2026-03-20 16:26:25.653266 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.653270 | controller | | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J\n logger.go:42: 2026-03-20 16:26:25.653284 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU\n 2026-03-20 16:26:25.653288 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.653293 | controller | | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq\n logger.go:42: 2026-03-20 16:26:25.653297 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6\n 2026-03-20 16:26:25.653303 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.653307 | controller | | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3\n logger.go:42: 2026-03-20 16:26:25.653311 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk\n 2026-03-20 16:26:25.653315 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.653320 | controller | | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb\n logger.go:42: 2026-03-20 16:26:25.653331 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE\n 2026-03-20 16:26:25.653336 | controller | \ logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.653340 | controller | | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==\n logger.go:42: 2026-03-20 16:26:25.653353 | controller | 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate | -----END 2026-03-20 16:26:25.653357 | controller | CERTIFICATE-----' ']'\n logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.653361 | controller | | test step completed 3-change-internal-svc-certificate\n logger.go:42: 16:19:12 2026-03-20 16:26:25.653365 | controller | | watcher-tls-certs-change/4-cleanup-watcher | starting test step 4-cleanup-watcher\n 2026-03-20 16:26:25.653369 | controller | \ logger.go:42: 16:19:19 | watcher-tls-certs-change/4-cleanup-watcher | test step 2026-03-20 16:26:25.653373 | controller | completed 4-cleanup-watcher\n logger.go:42: 16:19:19 | watcher-tls-certs-change/5-clenaup-certs 2026-03-20 16:26:25.653380 | controller | | starting test step 5-clenaup-certs\n logger.go:42: 16:19:19 | watcher-tls-certs-change/5-clenaup-certs 2026-03-20 16:26:25.653385 | controller | | test step completed 5-clenaup-certs\n logger.go:42: 16:19:19 | watcher-tls-certs-change 2026-03-20 16:26:25.653389 | controller | | skipping kubernetes event logging\n=== CONT kuttl/harness/watcher-tls\n logger.go:42: 2026-03-20 16:26:25.653393 | controller | 16:19:19 | watcher-tls | Skipping creation of user-supplied namespace: watcher-kuttl-default\n 2026-03-20 16:26:25.653397 | controller | \ logger.go:42: 16:19:19 | watcher-tls/0-cleanup-watcher | starting test step 2026-03-20 16:26:25.653401 | controller | 0-cleanup-watcher\n logger.go:42: 16:19:19 | watcher-tls/0-cleanup-watcher | 2026-03-20 16:26:25.653405 | controller | test step completed 0-cleanup-watcher\n logger.go:42: 16:19:19 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.653417 | controller | | starting test step 1-deploy-with-tlse\n logger.go:42: 16:19:19 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.653422 | controller | | Certificate:watcher-kuttl-default/watcher-internal-svc created\n logger.go:42: 2026-03-20 16:26:25.653426 | controller | 16:19:19 | watcher-tls/1-deploy-with-tlse | Certificate:watcher-kuttl-default/watcher-public-svc 2026-03-20 16:26:25.653430 | controller | created\n logger.go:42: 16:19:19 | watcher-tls/1-deploy-with-tlse | Watcher:watcher-kuttl-default/watcher-kuttl 2026-03-20 16:26:25.653434 | controller | created\n logger.go:42: 16:19:19 | watcher-tls/1-deploy-with-tlse | running command: 2026-03-20 16:26:25.653438 | controller | [sh -c set -euxo pipefail\n oc exec -n ${NAMESPACE} openstackclient -- openstack 2026-03-20 16:26:25.653443 | controller | service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc 2026-03-20 16:26:25.653447 | controller | exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name 2026-03-20 16:26:25.653451 | controller | -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n ${NAMESPACE} 2026-03-20 16:26:25.653455 | controller | keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ 2026-03-20 16:26:25.653459 | controller | -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" 2026-03-20 16:26:25.653467 | controller | ]\n [ \"$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o 2026-03-20 16:26:25.653471 | controller | jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" == 1 ]\n [ \"$(oc 2026-03-20 16:26:25.653475 | controller | get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64 2026-03-20 16:26:25.653481 | controller | -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" == 2 2026-03-20 16:26:25.653486 | controller | ]\n # check that both endpoints have https set\n oc exec -n ${NAMESPACE} 2026-03-20 16:26:25.653490 | controller | openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https) 2026-03-20 16:26:25.653494 | controller | == 2 ]\n # If we are running the container locally, skip following test\n 2026-03-20 16:26:25.653498 | controller | \ if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" 2026-03-20 16:26:25.653502 | controller | == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set 2026-03-20 16:26:25.653506 | controller | env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) 2026-03-20 16:26:25.653511 | controller | -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; 2026-03-20 16:26:25.653515 | controller | do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo 2026-03-20 16:26:25.653519 | controller | ${i}\n counter=$((counter + 1))\n fi\n done\n if 2026-03-20 16:26:25.653523 | controller | [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables 2026-03-20 16:26:25.653527 | controller | found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT 2026-03-20 16:26:25.653538 | controller | variables found.\"\n fi\n ]\n logger.go:42: 16:19:19 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.653545 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-20 16:26:25.653549 | controller | value -c Name -c Type\n logger.go:42: 16:19:19 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.653553 | controller | | ++ grep -c '^watcher'\n logger.go:42: 16:19:22 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.653557 | controller | | + '[' 0 == 1 ']'\n logger.go:42: 16:19:23 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.653562 | controller | | running command: [sh -c set -euxo pipefail\n oc exec -n ${NAMESPACE} openstackclient 2026-03-20 16:26:25.653566 | controller | -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n 2026-03-20 16:26:25.653570 | controller | \ SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-20 16:26:25.653574 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ 2026-03-20 16:26:25.653579 | controller | $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.653583 | controller | == ${SERVICEID} ]\n [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl 2026-03-20 16:26:25.653587 | controller | -o jsonpath={.status.hash.dbsync})\" ]\n [ \"$(oc get -n ${NAMESPACE} secret 2026-03-20 16:26:25.653591 | controller | watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" 2026-03-20 16:26:25.653595 | controller | == 1 ]\n [ \"$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.653599 | controller | -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" 2026-03-20 16:26:25.653603 | controller | == 2 ]\n # check that both endpoints have https set\n oc exec -n ${NAMESPACE} 2026-03-20 16:26:25.653608 | controller | openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https) 2026-03-20 16:26:25.653612 | controller | == 2 ]\n # If we are running the container locally, skip following test\n 2026-03-20 16:26:25.653619 | controller | \ if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" 2026-03-20 16:26:25.653623 | controller | == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set 2026-03-20 16:26:25.653627 | controller | env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) 2026-03-20 16:26:25.653631 | controller | -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; 2026-03-20 16:26:25.653635 | controller | do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo 2026-03-20 16:26:25.653647 | controller | ${i}\n counter=$((counter + 1))\n fi\n done\n if 2026-03-20 16:26:25.653652 | controller | [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables 2026-03-20 16:26:25.653656 | controller | found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT 2026-03-20 16:26:25.653660 | controller | variables found.\"\n fi\n ]\n logger.go:42: 16:19:23 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.653664 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-20 16:26:25.653668 | controller | value -c Name -c Type\n logger.go:42: 16:19:23 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.653673 | controller | | ++ grep -c '^watcher'\n logger.go:42: 16:19:25 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.653677 | controller | | + '[' 0 == 1 ']'\n logger.go:42: 16:19:26 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.653681 | controller | | running command: [sh -c set -euxo pipefail\n oc exec -n ${NAMESPACE} openstackclient 2026-03-20 16:26:25.653685 | controller | -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n 2026-03-20 16:26:25.653689 | controller | \ SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-20 16:26:25.653745 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ 2026-03-20 16:26:25.653753 | controller | $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.653757 | controller | == ${SERVICEID} ]\n [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl 2026-03-20 16:26:25.653762 | controller | -o jsonpath={.status.hash.dbsync})\" ]\n [ \"$(oc get -n ${NAMESPACE} secret 2026-03-20 16:26:25.653766 | controller | watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" 2026-03-20 16:26:25.653770 | controller | == 1 ]\n [ \"$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.653774 | controller | -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" 2026-03-20 16:26:25.653778 | controller | == 2 ]\n # check that both endpoints have https set\n oc exec -n ${NAMESPACE} 2026-03-20 16:26:25.653782 | controller | openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https) 2026-03-20 16:26:25.653787 | controller | == 2 ]\n # If we are running the container locally, skip following test\n 2026-03-20 16:26:25.653800 | controller | \ if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" 2026-03-20 16:26:25.653804 | controller | == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set 2026-03-20 16:26:25.653808 | controller | env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) 2026-03-20 16:26:25.653813 | controller | -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; 2026-03-20 16:26:25.653817 | controller | do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo 2026-03-20 16:26:25.653821 | controller | ${i}\n counter=$((counter + 1))\n fi\n done\n if 2026-03-20 16:26:25.653825 | controller | [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables 2026-03-20 16:26:25.653830 | controller | found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT 2026-03-20 16:26:25.653836 | controller | variables found.\"\n fi\n ]\n logger.go:42: 16:19:26 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.653840 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-20 16:26:25.653845 | controller | value -c Name -c Type\n logger.go:42: 16:19:26 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.653849 | controller | | ++ grep -c '^watcher'\n logger.go:42: 16:19:28 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.653853 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:19:28 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.653857 | controller | | ++ grep watcher\n logger.go:42: 16:19:28 | watcher-tls/1-deploy-with-tlse | 2026-03-20 16:26:25.653862 | controller | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-20 16:26:25.653866 | controller | value -c Name -c Type -c ID\n logger.go:42: 16:19:28 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.653870 | controller | | ++ awk '{print $1}'\n logger.go:42: 16:19:30 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.653874 | controller | | + SERVICEID=0bc8267151fd4fefbc0a3327396365a4\n logger.go:42: 16:19:30 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.653878 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-20 16:26:25.653882 | controller | \ logger.go:42: 16:19:30 | watcher-tls/1-deploy-with-tlse | + '[' 0bc8267151fd4fefbc0a3327396365a4 2026-03-20 16:26:25.653887 | controller | == 0bc8267151fd4fefbc0a3327396365a4 ']'\n logger.go:42: 16:19:30 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.653891 | controller | | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n 2026-03-20 16:26:25.653895 | controller | \ logger.go:42: 16:19:30 | watcher-tls/1-deploy-with-tlse | + '[' -n '' ']'\n 2026-03-20 16:26:25.653899 | controller | \ logger.go:42: 16:19:32 | watcher-tls/1-deploy-with-tlse | running command: [sh 2026-03-20 16:26:25.653903 | controller | -c set -euxo pipefail\n oc exec -n ${NAMESPACE} openstackclient -- openstack 2026-03-20 16:26:25.653908 | controller | service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc 2026-03-20 16:26:25.653913 | controller | exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name 2026-03-20 16:26:25.653918 | controller | -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n ${NAMESPACE} 2026-03-20 16:26:25.653922 | controller | keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ 2026-03-20 16:26:25.653926 | controller | -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" 2026-03-20 16:26:25.653930 | controller | ]\n [ \"$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o 2026-03-20 16:26:25.653934 | controller | jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" == 1 ]\n [ \"$(oc 2026-03-20 16:26:25.653938 | controller | get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64 2026-03-20 16:26:25.653943 | controller | -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" == 2 2026-03-20 16:26:25.653947 | controller | ]\n # check that both endpoints have https set\n oc exec -n ${NAMESPACE} 2026-03-20 16:26:25.653951 | controller | openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https) 2026-03-20 16:26:25.653955 | controller | == 2 ]\n # If we are running the container locally, skip following test\n 2026-03-20 16:26:25.653959 | controller | \ if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" 2026-03-20 16:26:25.653963 | controller | == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set 2026-03-20 16:26:25.653967 | controller | env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) 2026-03-20 16:26:25.653972 | controller | -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; 2026-03-20 16:26:25.653987 | controller | do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo 2026-03-20 16:26:25.653992 | controller | ${i}\n counter=$((counter + 1))\n fi\n done\n if 2026-03-20 16:26:25.654003 | controller | [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables 2026-03-20 16:26:25.654008 | controller | found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT 2026-03-20 16:26:25.654012 | controller | variables found.\"\n fi\n ]\n logger.go:42: 16:19:32 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.654016 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-20 16:26:25.654021 | controller | value -c Name -c Type\n logger.go:42: 16:19:32 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.654025 | controller | | ++ grep -c '^watcher'\n logger.go:42: 16:19:34 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.654029 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:19:34 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.654033 | controller | | ++ grep watcher\n logger.go:42: 16:19:34 | watcher-tls/1-deploy-with-tlse | 2026-03-20 16:26:25.654037 | controller | ++ awk '{print $1}'\n logger.go:42: 16:19:34 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.654041 | controller | | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.654045 | controller | -f value -c Name -c Type -c ID\n logger.go:42: 16:19:36 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.654050 | controller | | + SERVICEID=0bc8267151fd4fefbc0a3327396365a4\n logger.go:42: 16:19:36 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.654054 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-20 16:26:25.654060 | controller | \ logger.go:42: 16:19:36 | watcher-tls/1-deploy-with-tlse | + '[' 0bc8267151fd4fefbc0a3327396365a4 2026-03-20 16:26:25.654065 | controller | == 0bc8267151fd4fefbc0a3327396365a4 ']'\n logger.go:42: 16:19:36 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.654069 | controller | | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n 2026-03-20 16:26:25.654073 | controller | \ logger.go:42: 16:19:36 | watcher-tls/1-deploy-with-tlse | + '[' -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.654077 | controller | ']'\n logger.go:42: 16:19:36 | watcher-tls/1-deploy-with-tlse | ++ base64 -d\n 2026-03-20 16:26:25.654081 | controller | \ logger.go:42: 16:19:36 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.654086 | controller | secret watcher-kuttl-api-config-data -o 'jsonpath={.data.my\\.cnf}'\n logger.go:42: 2026-03-20 16:26:25.654090 | controller | 16:19:36 | watcher-tls/1-deploy-with-tlse | ++ grep -c ssl=1\n logger.go:42: 2026-03-20 16:26:25.654094 | controller | 16:19:36 | watcher-tls/1-deploy-with-tlse | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-20 16:26:25.654098 | controller | 16:19:36 | watcher-tls/1-deploy-with-tlse | ++ grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'\n 2026-03-20 16:26:25.654102 | controller | \ logger.go:42: 16:19:36 | watcher-tls/1-deploy-with-tlse | ++ base64 -d\n logger.go:42: 2026-03-20 16:26:25.654106 | controller | 16:19:36 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret 2026-03-20 16:26:25.654111 | controller | watcher-kuttl-api-config-data -o 'jsonpath={.data.00-default\\.conf}'\n logger.go:42: 2026-03-20 16:26:25.654115 | controller | 16:19:37 | watcher-tls/1-deploy-with-tlse | + '[' 2 == 2 ']'\n logger.go:42: 2026-03-20 16:26:25.654119 | controller | 16:19:37 | watcher-tls/1-deploy-with-tlse | + grep infra-optim\n logger.go:42: 2026-03-20 16:26:25.654123 | controller | 16:19:37 | watcher-tls/1-deploy-with-tlse | ++ grep -c https\n logger.go:42: 2026-03-20 16:26:25.654127 | controller | 16:19:37 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default openstackclient 2026-03-20 16:26:25.654135 | controller | -- openstack endpoint list\n logger.go:42: 16:19:39 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.654139 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 16:19:40 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.654144 | controller | | running command: [sh -c set -euxo pipefail\n oc exec -n ${NAMESPACE} openstackclient 2026-03-20 16:26:25.654148 | controller | -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n 2026-03-20 16:26:25.654152 | controller | \ SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-20 16:26:25.654156 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ 2026-03-20 16:26:25.654160 | controller | $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.654164 | controller | == ${SERVICEID} ]\n [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl 2026-03-20 16:26:25.654169 | controller | -o jsonpath={.status.hash.dbsync})\" ]\n [ \"$(oc get -n ${NAMESPACE} secret 2026-03-20 16:26:25.654173 | controller | watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" 2026-03-20 16:26:25.654177 | controller | == 1 ]\n [ \"$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.654181 | controller | -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" 2026-03-20 16:26:25.654193 | controller | == 2 ]\n # check that both endpoints have https set\n oc exec -n ${NAMESPACE} 2026-03-20 16:26:25.654197 | controller | openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https) 2026-03-20 16:26:25.654202 | controller | == 2 ]\n # If we are running the container locally, skip following test\n 2026-03-20 16:26:25.654206 | controller | \ if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" 2026-03-20 16:26:25.654210 | controller | == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set 2026-03-20 16:26:25.654214 | controller | env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) 2026-03-20 16:26:25.654218 | controller | -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; 2026-03-20 16:26:25.654222 | controller | do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo 2026-03-20 16:26:25.654227 | controller | ${i}\n counter=$((counter + 1))\n fi\n done\n if 2026-03-20 16:26:25.654231 | controller | [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables 2026-03-20 16:26:25.654235 | controller | found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT 2026-03-20 16:26:25.654239 | controller | variables found.\"\n fi\n ]\n logger.go:42: 16:19:40 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.654243 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-20 16:26:25.654247 | controller | value -c Name -c Type\n logger.go:42: 16:19:40 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.654252 | controller | | ++ grep -c '^watcher'\n logger.go:42: 16:19:42 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.654256 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:19:42 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.654260 | controller | | ++ grep watcher\n logger.go:42: 16:19:42 | watcher-tls/1-deploy-with-tlse | 2026-03-20 16:26:25.654264 | controller | ++ awk '{print $1}'\n logger.go:42: 16:19:42 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.654268 | controller | | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.654272 | controller | -f value -c Name -c Type -c ID\n logger.go:42: 16:19:44 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.654276 | controller | | + SERVICEID=0bc8267151fd4fefbc0a3327396365a4\n logger.go:42: 16:19:44 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.654282 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-20 16:26:25.654286 | controller | \ logger.go:42: 16:19:44 | watcher-tls/1-deploy-with-tlse | + '[' 0bc8267151fd4fefbc0a3327396365a4 2026-03-20 16:26:25.654290 | controller | == 0bc8267151fd4fefbc0a3327396365a4 ']'\n logger.go:42: 16:19:44 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.654295 | controller | | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n 2026-03-20 16:26:25.654301 | controller | \ logger.go:42: 16:19:44 | watcher-tls/1-deploy-with-tlse | + '[' -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.654306 | controller | ']'\n logger.go:42: 16:19:44 | watcher-tls/1-deploy-with-tlse | ++ base64 -d\n 2026-03-20 16:26:25.654310 | controller | \ logger.go:42: 16:19:44 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.654314 | controller | secret watcher-kuttl-api-config-data -o 'jsonpath={.data.my\\.cnf}'\n logger.go:42: 2026-03-20 16:26:25.654318 | controller | 16:19:44 | watcher-tls/1-deploy-with-tlse | ++ grep -c ssl=1\n logger.go:42: 2026-03-20 16:26:25.654322 | controller | 16:19:45 | watcher-tls/1-deploy-with-tlse | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-20 16:26:25.654326 | controller | 16:19:45 | watcher-tls/1-deploy-with-tlse | ++ base64 -d\n logger.go:42: 16:19:45 2026-03-20 16:26:25.654331 | controller | | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.654335 | controller | -o 'jsonpath={.data.00-default\\.conf}'\n logger.go:42: 16:19:45 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.654339 | controller | | ++ grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'\n logger.go:42: 2026-03-20 16:26:25.654343 | controller | 16:19:45 | watcher-tls/1-deploy-with-tlse | + '[' 2 == 2 ']'\n logger.go:42: 2026-03-20 16:26:25.654347 | controller | 16:19:45 | watcher-tls/1-deploy-with-tlse | + grep infra-optim\n logger.go:42: 2026-03-20 16:26:25.654352 | controller | 16:19:45 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default openstackclient 2026-03-20 16:26:25.654356 | controller | -- openstack endpoint list\n logger.go:42: 16:19:45 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.654362 | controller | | ++ grep -c https\n logger.go:42: 16:19:47 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.654366 | controller | | + '[' 2 == 2 ']'\n logger.go:42: 16:19:47 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.654370 | controller | | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n 2026-03-20 16:26:25.654382 | controller | \ logger.go:42: 16:19:47 | watcher-tls/1-deploy-with-tlse | + '[' '' == '' ']'\n 2026-03-20 16:26:25.654386 | controller | \ logger.go:42: 16:19:47 | watcher-tls/1-deploy-with-tlse | + exit 0\n logger.go:42: 2026-03-20 16:26:25.654390 | controller | 16:19:48 | watcher-tls/1-deploy-with-tlse | running command: [sh -c set -euxo pipefail\n 2026-03-20 16:26:25.654395 | controller | \ oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-20 16:26:25.654399 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc exec -n ${NAMESPACE} 2026-03-20 16:26:25.654403 | controller | openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep 2026-03-20 16:26:25.654407 | controller | watcher| awk '{print $1}')\n [ $(oc get -n ${NAMESPACE} keystoneservice watcher 2026-03-20 16:26:25.654411 | controller | -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ -n \"$(oc get -n ${NAMESPACE} 2026-03-20 16:26:25.654416 | controller | watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n [ \"$(oc get 2026-03-20 16:26:25.654420 | controller | -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 2026-03-20 16:26:25.654424 | controller | -d|grep -c 'ssl=1')\" == 1 ]\n [ \"$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.654428 | controller | -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" 2026-03-20 16:26:25.654436 | controller | == 2 ]\n # check that both endpoints have https set\n oc exec -n ${NAMESPACE} 2026-03-20 16:26:25.654440 | controller | openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https) 2026-03-20 16:26:25.654444 | controller | == 2 ]\n # If we are running the container locally, skip following test\n 2026-03-20 16:26:25.654450 | controller | \ if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" 2026-03-20 16:26:25.654454 | controller | == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set 2026-03-20 16:26:25.654458 | controller | env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) 2026-03-20 16:26:25.654463 | controller | -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; 2026-03-20 16:26:25.654467 | controller | do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo 2026-03-20 16:26:25.654471 | controller | ${i}\n counter=$((counter + 1))\n fi\n done\n if 2026-03-20 16:26:25.654475 | controller | [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables 2026-03-20 16:26:25.654479 | controller | found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT 2026-03-20 16:26:25.654484 | controller | variables found.\"\n fi\n ]\n logger.go:42: 16:19:48 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.654488 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-20 16:26:25.654492 | controller | value -c Name -c Type\n logger.go:42: 16:19:48 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.654496 | controller | | ++ grep -c '^watcher'\n logger.go:42: 16:19:50 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.654500 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:19:50 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.654504 | controller | | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.654508 | controller | -f value -c Name -c Type -c ID\n logger.go:42: 16:19:50 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.654513 | controller | | ++ grep watcher\n logger.go:42: 16:19:50 | watcher-tls/1-deploy-with-tlse | 2026-03-20 16:26:25.654517 | controller | ++ awk '{print $1}'\n logger.go:42: 16:19:52 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.654521 | controller | | + SERVICEID=0bc8267151fd4fefbc0a3327396365a4\n logger.go:42: 16:19:52 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.654525 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-20 16:26:25.654529 | controller | \ logger.go:42: 16:19:52 | watcher-tls/1-deploy-with-tlse | + '[' 0bc8267151fd4fefbc0a3327396365a4 2026-03-20 16:26:25.654534 | controller | == 0bc8267151fd4fefbc0a3327396365a4 ']'\n logger.go:42: 16:19:52 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.654538 | controller | | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n 2026-03-20 16:26:25.654542 | controller | \ logger.go:42: 16:19:52 | watcher-tls/1-deploy-with-tlse | + '[' -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.654546 | controller | ']'\n logger.go:42: 16:19:52 | watcher-tls/1-deploy-with-tlse | ++ base64 -d\n 2026-03-20 16:26:25.654550 | controller | \ logger.go:42: 16:19:52 | watcher-tls/1-deploy-with-tlse | ++ grep -c ssl=1\n 2026-03-20 16:26:25.654554 | controller | \ logger.go:42: 16:19:52 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.654558 | controller | secret watcher-kuttl-api-config-data -o 'jsonpath={.data.my\\.cnf}'\n logger.go:42: 2026-03-20 16:26:25.654570 | controller | 16:19:52 | watcher-tls/1-deploy-with-tlse | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-20 16:26:25.654576 | controller | 16:19:52 | watcher-tls/1-deploy-with-tlse | ++ grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'\n 2026-03-20 16:26:25.654580 | controller | \ logger.go:42: 16:19:52 | watcher-tls/1-deploy-with-tlse | ++ base64 -d\n logger.go:42: 2026-03-20 16:26:25.654585 | controller | 16:19:52 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default secret 2026-03-20 16:26:25.654589 | controller | watcher-kuttl-api-config-data -o 'jsonpath={.data.00-default\\.conf}'\n logger.go:42: 2026-03-20 16:26:25.654593 | controller | 16:19:53 | watcher-tls/1-deploy-with-tlse | + '[' 2 == 2 ']'\n logger.go:42: 2026-03-20 16:26:25.654597 | controller | 16:19:53 | watcher-tls/1-deploy-with-tlse | + grep infra-optim\n logger.go:42: 2026-03-20 16:26:25.654601 | controller | 16:19:53 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default openstackclient 2026-03-20 16:26:25.654607 | controller | -- openstack endpoint list\n logger.go:42: 16:19:53 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.654611 | controller | | ++ grep -c https\n logger.go:42: 16:19:55 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.654616 | controller | | + '[' 2 == 2 ']'\n logger.go:42: 16:19:55 | watcher-tls/1-deploy-with-tlse 2026-03-20 16:26:25.654620 | controller | | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n 2026-03-20 16:26:25.654624 | controller | \ logger.go:42: 16:19:55 | watcher-tls/1-deploy-with-tlse | + '[' '' == '' ']'\n 2026-03-20 16:26:25.654628 | controller | \ logger.go:42: 16:19:55 | watcher-tls/1-deploy-with-tlse | + exit 0\n logger.go:42: 2026-03-20 16:26:25.654632 | controller | 16:19:55 | watcher-tls/1-deploy-with-tlse | test step completed 1-deploy-with-tlse\n 2026-03-20 16:26:25.654636 | controller | \ logger.go:42: 16:19:55 | watcher-tls/2-patch-mtls | starting test step 2-patch-mtls\n 2026-03-20 16:26:25.654641 | controller | \ logger.go:42: 16:19:55 | watcher-tls/2-patch-mtls | running command: [sh -c 2026-03-20 16:26:25.654645 | controller | set -euxo pipefail\n oc patch oscp -n $NAMESPACE openstack --type='json' 2026-03-20 16:26:25.654649 | controller | -p='[{\"op\": \"replace\", \"path\": \"/spec/memcached/templates/memcached/tls/mtls/sslVerifyMode\", 2026-03-20 16:26:25.654653 | controller | \"value\": \"Request\"}]'\n ]\n logger.go:42: 16:19:55 | watcher-tls/2-patch-mtls 2026-03-20 16:26:25.654657 | controller | | + oc patch oscp -n watcher-kuttl-default openstack --type=json '-p=[{\"op\": \"replace\", 2026-03-20 16:26:25.654661 | controller | \"path\": \"/spec/memcached/templates/memcached/tls/mtls/sslVerifyMode\", \"value\": 2026-03-20 16:26:25.654665 | controller | \"Request\"}]'\n logger.go:42: 16:19:55 | watcher-tls/2-patch-mtls | openstackcontrolplane.core.openstack.org/openstack 2026-03-20 16:26:25.654670 | controller | patched\n logger.go:42: 16:19:55 | watcher-tls/2-patch-mtls | running command: 2026-03-20 16:26:25.654674 | controller | [sh -c set -euxo pipefail\n \n oc project ${NAMESPACE}\n # 2026-03-20 16:26:25.654678 | controller | Get pod names for each watcher service\n APIPOD=$(oc get pods -l service=watcher-api 2026-03-20 16:26:25.654682 | controller | -o jsonpath='{.items[0].metadata.name}')\n APPLIERPOD=$(oc get pods -l service=watcher-applier 2026-03-20 16:26:25.654686 | controller | -o jsonpath='{.items[0].metadata.name}')\n DECISIONENGINEPOD=$(oc get pods 2026-03-20 16:26:25.654690 | controller | -l service=watcher-decision-engine -o jsonpath='{.items[0].metadata.name}')\n \n 2026-03-20 16:26:25.654708 | controller | \ # Verify memcached mTLS config parameters in watcher-api config\n if 2026-03-20 16:26:25.654713 | controller | [ -n \"${APIPOD}\" ]; then\n echo \"Checking watcher-api config...\"\n 2026-03-20 16:26:25.654717 | controller | \ [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.654721 | controller | | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ 2026-03-20 16:26:25.654725 | controller | $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.654731 | controller | | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ 2026-03-20 16:26:25.654735 | controller | $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.654739 | controller | | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ 2026-03-20 16:26:25.654743 | controller | $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.654748 | controller | | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n # Verify 2026-03-20 16:26:25.654752 | controller | mTLS config parameters in memcached backend config\n [ $(echo $(oc rsh 2026-03-20 16:26:25.654756 | controller | -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c 2026-03-20 16:26:25.654760 | controller | \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo $(oc rsh 2026-03-20 16:26:25.654764 | controller | -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c 2026-03-20 16:26:25.654768 | controller | \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo $(oc rsh 2026-03-20 16:26:25.654781 | controller | -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c 2026-03-20 16:26:25.654785 | controller | \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n exit 2026-03-20 16:26:25.654790 | controller | 1\n fi\n \n # Verify memcached mTLS config parameters in watcher-applier 2026-03-20 16:26:25.654794 | controller | config\n if [ -n \"${APPLIERPOD}\" ]; then\n echo \"Checking watcher-applier 2026-03-20 16:26:25.654798 | controller | config...\"\n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.654802 | controller | | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ 2026-03-20 16:26:25.654806 | controller | $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.654811 | controller | | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ 2026-03-20 16:26:25.654815 | controller | $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.654821 | controller | | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ 2026-03-20 16:26:25.654825 | controller | $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.654829 | controller | | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n [ $(echo 2026-03-20 16:26:25.654834 | controller | $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.654838 | controller | | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo 2026-03-20 16:26:25.654842 | controller | $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.654846 | controller | | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo 2026-03-20 16:26:25.654850 | controller | $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.654854 | controller | | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n 2026-03-20 16:26:25.654858 | controller | \ exit 1\n fi\n \n # Verify memcached mTLS config parameters 2026-03-20 16:26:25.654863 | controller | in watcher-decision-engine config\n if [ -n \"${DECISIONENGINEPOD}\" ]; then\n 2026-03-20 16:26:25.654867 | controller | \ echo \"Checking watcher-decision-engine config...\"\n [ $(echo 2026-03-20 16:26:25.654871 | controller | $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.654875 | controller | | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ 2026-03-20 16:26:25.654879 | controller | $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.654885 | controller | | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ 2026-03-20 16:26:25.654889 | controller | $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.654893 | controller | | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ 2026-03-20 16:26:25.654897 | controller | $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.654902 | controller | | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n [ $(echo 2026-03-20 16:26:25.654906 | controller | $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.654910 | controller | | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo 2026-03-20 16:26:25.654922 | controller | $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.654927 | controller | | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo 2026-03-20 16:26:25.654931 | controller | $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.654935 | controller | | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n 2026-03-20 16:26:25.654940 | controller | \ exit 1\n fi\n ]\n logger.go:42: 16:19:55 | watcher-tls/2-patch-mtls 2026-03-20 16:26:25.654944 | controller | | + oc project watcher-kuttl-default\n logger.go:42: 16:19:55 | watcher-tls/2-patch-mtls 2026-03-20 16:26:25.654948 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.654952 | controller | \ logger.go:42: 16:19:55 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api 2026-03-20 16:26:25.654956 | controller | -o 'jsonpath={.items[0].metadata.name}'\n logger.go:42: 16:19:55 | watcher-tls/2-patch-mtls 2026-03-20 16:26:25.654961 | controller | | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 16:19:55 | watcher-tls/2-patch-mtls 2026-03-20 16:26:25.654965 | controller | | ++ oc get pods -l service=watcher-applier -o 'jsonpath={.items[0].metadata.name}'\n 2026-03-20 16:26:25.654977 | controller | \ logger.go:42: 16:19:55 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0\n 2026-03-20 16:26:25.654981 | controller | \ logger.go:42: 16:19:55 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine 2026-03-20 16:26:25.654985 | controller | -o 'jsonpath={.items[0].metadata.name}'\n logger.go:42: 16:19:56 | watcher-tls/2-patch-mtls 2026-03-20 16:26:25.654990 | controller | | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0\n logger.go:42: 16:19:56 2026-03-20 16:26:25.654994 | controller | | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 2026-03-20 16:26:25.654998 | controller | 16:19:56 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-api config...'\n 2026-03-20 16:26:25.655002 | controller | \ logger.go:42: 16:19:56 | watcher-tls/2-patch-mtls | Checking watcher-api config...\n 2026-03-20 16:26:25.655006 | controller | \ logger.go:42: 16:19:56 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile 2026-03-20 16:26:25.655011 | controller | = /etc/pki/tls/certs/mtls.crt '\n logger.go:42: 16:19:56 | watcher-tls/2-patch-mtls 2026-03-20 16:26:25.655015 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-20 16:26:25.655019 | controller | \ logger.go:42: 16:19:56 | watcher-tls/2-patch-mtls | error: Internal error occurred: 2026-03-20 16:26:25.655023 | controller | error executing command in container: container is not created or running\n logger.go:42: 2026-03-20 16:26:25.655028 | controller | 16:19:56 | watcher-tls/2-patch-mtls | ++ echo\n logger.go:42: 16:19:56 | watcher-tls/2-patch-mtls 2026-03-20 16:26:25.655032 | controller | | + '[' 0 == 1 ']'\n logger.go:42: 16:19:57 | watcher-tls/2-patch-mtls | running 2026-03-20 16:26:25.655038 | controller | command: [sh -c set -euxo pipefail\n \n oc project ${NAMESPACE}\n 2026-03-20 16:26:25.655042 | controller | \ # Get pod names for each watcher service\n APIPOD=$(oc get pods -l 2026-03-20 16:26:25.655047 | controller | service=watcher-api -o jsonpath='{.items[0].metadata.name}')\n APPLIERPOD=$(oc 2026-03-20 16:26:25.655051 | controller | get pods -l service=watcher-applier -o jsonpath='{.items[0].metadata.name}')\n DECISIONENGINEPOD=$(oc 2026-03-20 16:26:25.655055 | controller | get pods -l service=watcher-decision-engine -o jsonpath='{.items[0].metadata.name}')\n 2026-03-20 16:26:25.655059 | controller | \ \n # Verify memcached mTLS config parameters in watcher-api config\n 2026-03-20 16:26:25.655063 | controller | \ if [ -n \"${APIPOD}\" ]; then\n echo \"Checking watcher-api config...\"\n 2026-03-20 16:26:25.655067 | controller | \ [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.655074 | controller | | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ 2026-03-20 16:26:25.655078 | controller | $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.655082 | controller | | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ 2026-03-20 16:26:25.655086 | controller | $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.655090 | controller | | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ 2026-03-20 16:26:25.655094 | controller | $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.655099 | controller | | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n # Verify 2026-03-20 16:26:25.655103 | controller | mTLS config parameters in memcached backend config\n [ $(echo $(oc rsh 2026-03-20 16:26:25.655107 | controller | -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c 2026-03-20 16:26:25.655111 | controller | \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo $(oc rsh 2026-03-20 16:26:25.655115 | controller | -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c 2026-03-20 16:26:25.655130 | controller | \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo $(oc rsh 2026-03-20 16:26:25.655134 | controller | -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c 2026-03-20 16:26:25.655138 | controller | \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n exit 2026-03-20 16:26:25.655142 | controller | 1\n fi\n \n # Verify memcached mTLS config parameters in watcher-applier 2026-03-20 16:26:25.655146 | controller | config\n if [ -n \"${APPLIERPOD}\" ]; then\n echo \"Checking watcher-applier 2026-03-20 16:26:25.655151 | controller | config...\"\n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.655155 | controller | | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ 2026-03-20 16:26:25.655159 | controller | $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.655163 | controller | | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ 2026-03-20 16:26:25.655174 | controller | $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.655178 | controller | | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ 2026-03-20 16:26:25.655183 | controller | $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.655195 | controller | | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n [ $(echo 2026-03-20 16:26:25.655199 | controller | $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.655206 | controller | | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo 2026-03-20 16:26:25.655226 | controller | $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.655230 | controller | | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo 2026-03-20 16:26:25.655234 | controller | $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.655238 | controller | | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n 2026-03-20 16:26:25.655242 | controller | \ exit 1\n fi\n \n # Verify memcached mTLS config parameters 2026-03-20 16:26:25.655247 | controller | in watcher-decision-engine config\n if [ -n \"${DECISIONENGINEPOD}\" ]; then\n 2026-03-20 16:26:25.655251 | controller | \ echo \"Checking watcher-decision-engine config...\"\n [ $(echo 2026-03-20 16:26:25.655255 | controller | $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.655259 | controller | | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ 2026-03-20 16:26:25.655271 | controller | $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.655276 | controller | | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ 2026-03-20 16:26:25.655282 | controller | $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.655286 | controller | | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ 2026-03-20 16:26:25.655291 | controller | $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.655295 | controller | | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n [ $(echo 2026-03-20 16:26:25.655299 | controller | $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.655303 | controller | | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo 2026-03-20 16:26:25.655307 | controller | $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.655313 | controller | | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo 2026-03-20 16:26:25.655317 | controller | $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.655321 | controller | | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n 2026-03-20 16:26:25.655325 | controller | \ exit 1\n fi\n ]\n logger.go:42: 16:19:57 | watcher-tls/2-patch-mtls 2026-03-20 16:26:25.655330 | controller | | + oc project watcher-kuttl-default\n logger.go:42: 16:19:57 | watcher-tls/2-patch-mtls 2026-03-20 16:26:25.655334 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.655340 | controller | \ logger.go:42: 16:19:57 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api 2026-03-20 16:26:25.655345 | controller | -o 'jsonpath={.items[0].metadata.name}'\n logger.go:42: 16:19:57 | watcher-tls/2-patch-mtls 2026-03-20 16:26:25.655349 | controller | | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 16:19:57 | watcher-tls/2-patch-mtls 2026-03-20 16:26:25.655353 | controller | | ++ oc get pods -l service=watcher-applier -o 'jsonpath={.items[0].metadata.name}'\n 2026-03-20 16:26:25.655358 | controller | \ logger.go:42: 16:19:57 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0\n 2026-03-20 16:26:25.655362 | controller | \ logger.go:42: 16:19:57 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine 2026-03-20 16:26:25.655366 | controller | -o 'jsonpath={.items[0].metadata.name}'\n logger.go:42: 16:19:58 | watcher-tls/2-patch-mtls 2026-03-20 16:26:25.655372 | controller | | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0\n logger.go:42: 16:19:58 2026-03-20 16:26:25.655376 | controller | | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 2026-03-20 16:26:25.655380 | controller | 16:19:58 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-api config...'\n 2026-03-20 16:26:25.655385 | controller | \ logger.go:42: 16:19:58 | watcher-tls/2-patch-mtls | Checking watcher-api config...\n 2026-03-20 16:26:25.655389 | controller | \ logger.go:42: 16:19:58 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile 2026-03-20 16:26:25.655393 | controller | = /etc/pki/tls/certs/mtls.crt '\n logger.go:42: 16:19:58 | watcher-tls/2-patch-mtls 2026-03-20 16:26:25.655404 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-20 16:26:25.655409 | controller | \ logger.go:42: 16:19:58 | watcher-tls/2-patch-mtls | error: unable to upgrade 2026-03-20 16:26:25.655413 | controller | connection: container not found (\"watcher-api\")\n logger.go:42: 16:19:58 | 2026-03-20 16:26:25.655417 | controller | watcher-tls/2-patch-mtls | ++ echo\n logger.go:42: 16:19:58 | watcher-tls/2-patch-mtls 2026-03-20 16:26:25.655421 | controller | | + '[' 0 == 1 ']'\n logger.go:42: 16:19:59 | watcher-tls/2-patch-mtls | running 2026-03-20 16:26:25.655425 | controller | command: [sh -c set -euxo pipefail\n \n oc project ${NAMESPACE}\n 2026-03-20 16:26:25.655430 | controller | \ # Get pod names for each watcher service\n APIPOD=$(oc get pods -l 2026-03-20 16:26:25.655434 | controller | service=watcher-api -o jsonpath='{.items[0].metadata.name}')\n APPLIERPOD=$(oc 2026-03-20 16:26:25.655438 | controller | get pods -l service=watcher-applier -o jsonpath='{.items[0].metadata.name}')\n DECISIONENGINEPOD=$(oc 2026-03-20 16:26:25.655442 | controller | get pods -l service=watcher-decision-engine -o jsonpath='{.items[0].metadata.name}')\n 2026-03-20 16:26:25.655446 | controller | \ \n # Verify memcached mTLS config parameters in watcher-api config\n 2026-03-20 16:26:25.655450 | controller | \ if [ -n \"${APIPOD}\" ]; then\n echo \"Checking watcher-api config...\"\n 2026-03-20 16:26:25.655454 | controller | \ [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.655459 | controller | | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ 2026-03-20 16:26:25.655463 | controller | $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.655467 | controller | | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ 2026-03-20 16:26:25.655471 | controller | $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.655475 | controller | | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ 2026-03-20 16:26:25.655479 | controller | $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.655484 | controller | | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n # Verify 2026-03-20 16:26:25.655488 | controller | mTLS config parameters in memcached backend config\n [ $(echo $(oc rsh 2026-03-20 16:26:25.655492 | controller | -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c 2026-03-20 16:26:25.655496 | controller | \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo $(oc rsh 2026-03-20 16:26:25.655500 | controller | -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c 2026-03-20 16:26:25.655504 | controller | \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo $(oc rsh 2026-03-20 16:26:25.655508 | controller | -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) | grep -c 2026-03-20 16:26:25.655513 | controller | \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n exit 2026-03-20 16:26:25.655517 | controller | 1\n fi\n \n # Verify memcached mTLS config parameters in watcher-applier 2026-03-20 16:26:25.655524 | controller | config\n if [ -n \"${APPLIERPOD}\" ]; then\n echo \"Checking watcher-applier 2026-03-20 16:26:25.655528 | controller | config...\"\n [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.655535 | controller | | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ 2026-03-20 16:26:25.655539 | controller | $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.655544 | controller | | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ 2026-03-20 16:26:25.655577 | controller | $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.655582 | controller | | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ 2026-03-20 16:26:25.655587 | controller | $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.655592 | controller | | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n [ $(echo 2026-03-20 16:26:25.655597 | controller | $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.655602 | controller | | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo 2026-03-20 16:26:25.655607 | controller | $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.655612 | controller | | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo 2026-03-20 16:26:25.655617 | controller | $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.655622 | controller | | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n 2026-03-20 16:26:25.655635 | controller | \ exit 1\n fi\n \n # Verify memcached mTLS config parameters 2026-03-20 16:26:25.655640 | controller | in watcher-decision-engine config\n if [ -n \"${DECISIONENGINEPOD}\" ]; then\n 2026-03-20 16:26:25.655645 | controller | \ echo \"Checking watcher-decision-engine config...\"\n [ $(echo 2026-03-20 16:26:25.655650 | controller | $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.655655 | controller | | grep -c \"memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt \") == 1 ]\n [ 2026-03-20 16:26:25.655660 | controller | $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.655665 | controller | | grep -c \"memcache_tls_keyfile = /etc/pki/tls/private/mtls.key\") == 1 ]\n [ 2026-03-20 16:26:25.655670 | controller | $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.655675 | controller | | grep -c \"memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n [ 2026-03-20 16:26:25.655680 | controller | $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.655685 | controller | | grep -c \"memcache_tls_enabled = true\") == 1 ]\n \n [ $(echo 2026-03-20 16:26:25.655709 | controller | $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.655717 | controller | | grep -c \"tls_certfile=/etc/pki/tls/certs/mtls.crt\") == 1 ]\n [ $(echo 2026-03-20 16:26:25.655722 | controller | $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.655727 | controller | | grep -c \"tls_keyfile=/etc/pki/tls/private/mtls.key\") == 1 ]\n [ $(echo 2026-03-20 16:26:25.655732 | controller | $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.655737 | controller | | grep -c \"tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\") == 1 ]\n else\n 2026-03-20 16:26:25.655744 | controller | \ exit 1\n fi\n ]\n logger.go:42: 16:19:59 | watcher-tls/2-patch-mtls 2026-03-20 16:26:25.655749 | controller | | + oc project watcher-kuttl-default\n logger.go:42: 16:19:59 | watcher-tls/2-patch-mtls 2026-03-20 16:26:25.655754 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.655759 | controller | \ logger.go:42: 16:19:59 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api 2026-03-20 16:26:25.655764 | controller | -o 'jsonpath={.items[0].metadata.name}'\n logger.go:42: 16:19:59 | watcher-tls/2-patch-mtls 2026-03-20 16:26:25.655769 | controller | | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 16:19:59 | watcher-tls/2-patch-mtls 2026-03-20 16:26:25.655774 | controller | | ++ oc get pods -l service=watcher-applier -o 'jsonpath={.items[0].metadata.name}'\n 2026-03-20 16:26:25.655778 | controller | \ logger.go:42: 16:19:59 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0\n 2026-03-20 16:26:25.655783 | controller | \ logger.go:42: 16:19:59 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine 2026-03-20 16:26:25.655788 | controller | -o 'jsonpath={.items[0].metadata.name}'\n logger.go:42: 16:19:59 | watcher-tls/2-patch-mtls 2026-03-20 16:26:25.655793 | controller | | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0\n logger.go:42: 16:19:59 2026-03-20 16:26:25.655798 | controller | | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 2026-03-20 16:26:25.655803 | controller | 16:19:59 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-api config...'\n 2026-03-20 16:26:25.655808 | controller | \ logger.go:42: 16:19:59 | watcher-tls/2-patch-mtls | Checking watcher-api config...\n 2026-03-20 16:26:25.655812 | controller | \ logger.go:42: 16:19:59 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile 2026-03-20 16:26:25.655816 | controller | = /etc/pki/tls/certs/mtls.crt '\n logger.go:42: 16:19:59 | watcher-tls/2-patch-mtls 2026-03-20 16:26:25.655821 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-20 16:26:25.655825 | controller | \ logger.go:42: 16:20:00 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path 2026-03-20 16:26:25.655829 | controller | = /var/lib/watcher transport_url = 'rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.655835 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.655840 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.655844 | controller | = '[database]' connection = 'mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-20 16:26:25.655848 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-20 16:26:25.655860 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.655866 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.655870 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.655874 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.655878 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.655883 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.655889 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.655893 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.655899 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-20 16:26:25.655903 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-20 16:26:25.655907 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-20 16:26:25.655911 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-20 16:26:25.655916 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-20 16:26:25.655920 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.655924 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.655928 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.655932 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-20 16:26:25.655936 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.655941 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type 2026-03-20 16:26:25.655945 | controller | = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.655949 | controller | = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' 2026-03-20 16:26:25.655953 | controller | endpoint_type = internal region_name = regionOne '[keystone_client]' interface = 2026-03-20 16:26:25.655957 | controller | internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.655962 | controller | = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' 2026-03-20 16:26:25.655966 | controller | interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-20 16:26:25.655970 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-20 16:26:25.655974 | controller | period = 900\n logger.go:42: 16:20:00 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-20 16:26:25.655978 | controller | 1 ']'\n logger.go:42: 16:20:00 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_keyfile 2026-03-20 16:26:25.655983 | controller | = /etc/pki/tls/private/mtls.key'\n logger.go:42: 16:20:00 | watcher-tls/2-patch-mtls 2026-03-20 16:26:25.655987 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-20 16:26:25.655991 | controller | \ logger.go:42: 16:20:00 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path 2026-03-20 16:26:25.655995 | controller | = /var/lib/watcher transport_url = 'rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.655999 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.656004 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.656008 | controller | = '[database]' connection = 'mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-20 16:26:25.656012 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-20 16:26:25.656016 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.656020 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.656025 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.656030 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.656035 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.656047 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.656051 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.656056 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.656061 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-20 16:26:25.656067 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-20 16:26:25.656073 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-20 16:26:25.656079 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-20 16:26:25.656084 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-20 16:26:25.656090 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.656096 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.656101 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.656105 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-20 16:26:25.656109 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.656113 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type 2026-03-20 16:26:25.656117 | controller | = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.656122 | controller | = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' 2026-03-20 16:26:25.656126 | controller | endpoint_type = internal region_name = regionOne '[keystone_client]' interface = 2026-03-20 16:26:25.656130 | controller | internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.656134 | controller | = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' 2026-03-20 16:26:25.656138 | controller | interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-20 16:26:25.656143 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-20 16:26:25.656147 | controller | period = 900\n logger.go:42: 16:20:00 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-20 16:26:25.656151 | controller | 1 ']'\n logger.go:42: 16:20:00 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_cafile 2026-03-20 16:26:25.656155 | controller | = /etc/pki/tls/certs/mtls-ca.crt'\n logger.go:42: 16:20:00 | watcher-tls/2-patch-mtls 2026-03-20 16:26:25.656160 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-20 16:26:25.656166 | controller | \ logger.go:42: 16:20:00 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path 2026-03-20 16:26:25.656170 | controller | = /var/lib/watcher transport_url = 'rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.656175 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.656179 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.656183 | controller | = '[database]' connection = 'mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-20 16:26:25.656189 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-20 16:26:25.656197 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.656202 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.656208 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.656213 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.656218 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.656223 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.656228 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.656232 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.656236 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-20 16:26:25.656240 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-20 16:26:25.656254 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-20 16:26:25.656259 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-20 16:26:25.656263 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-20 16:26:25.656267 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.656271 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.656276 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.656280 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-20 16:26:25.656284 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.656288 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type 2026-03-20 16:26:25.656292 | controller | = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.656296 | controller | = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' 2026-03-20 16:26:25.656301 | controller | endpoint_type = internal region_name = regionOne '[keystone_client]' interface = 2026-03-20 16:26:25.656305 | controller | internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.656309 | controller | = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' 2026-03-20 16:26:25.656313 | controller | interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-20 16:26:25.656317 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-20 16:26:25.656322 | controller | period = 900\n logger.go:42: 16:20:00 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-20 16:26:25.656326 | controller | 1 ']'\n logger.go:42: 16:20:00 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_enabled 2026-03-20 16:26:25.656330 | controller | = true'\n logger.go:42: 16:20:00 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api 2026-03-20 16:26:25.656334 | controller | watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 2026-03-20 16:26:25.656338 | controller | 16:20:00 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher 2026-03-20 16:26:25.656345 | controller | transport_url = 'rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.656349 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.656354 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.656358 | controller | = '[database]' connection = 'mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-20 16:26:25.656362 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-20 16:26:25.656366 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.656370 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.656375 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.656379 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.656383 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.656387 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.656391 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.656395 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.656399 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-20 16:26:25.656404 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-20 16:26:25.656408 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-20 16:26:25.656412 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-20 16:26:25.656416 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-20 16:26:25.656420 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.656424 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.656436 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.656441 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-20 16:26:25.656445 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.656451 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type 2026-03-20 16:26:25.656455 | controller | = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.656460 | controller | = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' 2026-03-20 16:26:25.656464 | controller | endpoint_type = internal region_name = regionOne '[keystone_client]' interface = 2026-03-20 16:26:25.656468 | controller | internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.656472 | controller | = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' 2026-03-20 16:26:25.656476 | controller | interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-20 16:26:25.656481 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-20 16:26:25.656485 | controller | period = 900\n logger.go:42: 16:20:00 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-20 16:26:25.656491 | controller | 1 ']'\n logger.go:42: 16:20:00 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt\n 2026-03-20 16:26:25.656495 | controller | \ logger.go:42: 16:20:00 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api 2026-03-20 16:26:25.656499 | controller | watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 2026-03-20 16:26:25.656503 | controller | 16:20:01 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher 2026-03-20 16:26:25.656507 | controller | transport_url = 'rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.656512 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.656516 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.656520 | controller | = '[database]' connection = 'mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-20 16:26:25.656524 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-20 16:26:25.656529 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.656533 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.656537 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.656541 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.656545 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.656550 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.656554 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.656558 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.656563 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-20 16:26:25.656568 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-20 16:26:25.656574 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-20 16:26:25.656579 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-20 16:26:25.656585 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-20 16:26:25.656591 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.656596 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.656604 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.656609 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-20 16:26:25.656613 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.656617 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type 2026-03-20 16:26:25.656622 | controller | = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.656634 | controller | = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' 2026-03-20 16:26:25.656639 | controller | endpoint_type = internal region_name = regionOne '[keystone_client]' interface = 2026-03-20 16:26:25.656643 | controller | internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.656650 | controller | = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' 2026-03-20 16:26:25.656654 | controller | interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-20 16:26:25.656659 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-20 16:26:25.656663 | controller | period = 900\n logger.go:42: 16:20:01 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-20 16:26:25.656667 | controller | 1 ']'\n logger.go:42: 16:20:01 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key\n 2026-03-20 16:26:25.656671 | controller | \ logger.go:42: 16:20:01 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api 2026-03-20 16:26:25.656675 | controller | watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 2026-03-20 16:26:25.656680 | controller | 16:20:01 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher 2026-03-20 16:26:25.656684 | controller | transport_url = 'rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.656688 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.656704 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.656711 | controller | = '[database]' connection = 'mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-20 16:26:25.656715 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-20 16:26:25.656719 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.656724 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.656728 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.656732 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.656736 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.656740 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.656744 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.656748 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.656753 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-20 16:26:25.656806 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-20 16:26:25.656811 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-20 16:26:25.656815 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-20 16:26:25.656819 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-20 16:26:25.656824 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.656828 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.656832 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.656837 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-20 16:26:25.656842 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.656848 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type 2026-03-20 16:26:25.656852 | controller | = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.656856 | controller | = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' 2026-03-20 16:26:25.656860 | controller | endpoint_type = internal region_name = regionOne '[keystone_client]' interface = 2026-03-20 16:26:25.656865 | controller | internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.656869 | controller | = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' 2026-03-20 16:26:25.656873 | controller | interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-20 16:26:25.656886 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-20 16:26:25.656892 | controller | period = 900\n logger.go:42: 16:20:01 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-20 16:26:25.656898 | controller | 1 ']'\n logger.go:42: 16:20:01 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\n 2026-03-20 16:26:25.656903 | controller | \ logger.go:42: 16:20:01 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api 2026-03-20 16:26:25.656909 | controller | watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 2026-03-20 16:26:25.656913 | controller | 16:20:01 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher 2026-03-20 16:26:25.656918 | controller | transport_url = 'rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.656922 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.656926 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.656930 | controller | = '[database]' connection = 'mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-20 16:26:25.656935 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-20 16:26:25.656939 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.656943 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.656947 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.656964 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.656968 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.656973 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.656977 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.656981 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.656985 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-20 16:26:25.656990 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-20 16:26:25.656994 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-20 16:26:25.656998 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-20 16:26:25.657002 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-20 16:26:25.657008 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.657013 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.657019 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.657025 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-20 16:26:25.657031 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.657036 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type 2026-03-20 16:26:25.657042 | controller | = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.657048 | controller | = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' 2026-03-20 16:26:25.657052 | controller | endpoint_type = internal region_name = regionOne '[keystone_client]' interface = 2026-03-20 16:26:25.657056 | controller | internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.657060 | controller | = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' 2026-03-20 16:26:25.657065 | controller | interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-20 16:26:25.657069 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-20 16:26:25.657082 | controller | period = 900\n logger.go:42: 16:20:01 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-20 16:26:25.657086 | controller | 1 ']'\n logger.go:42: 16:20:01 | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-applier-0 2026-03-20 16:26:25.657093 | controller | ']'\n logger.go:42: 16:20:01 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-applier 2026-03-20 16:26:25.657106 | controller | config...'\n logger.go:42: 16:20:01 | watcher-tls/2-patch-mtls | Checking watcher-applier 2026-03-20 16:26:25.657111 | controller | config...\n logger.go:42: 16:20:01 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile 2026-03-20 16:26:25.657115 | controller | = /etc/pki/tls/certs/mtls.crt '\n logger.go:42: 16:20:01 | watcher-tls/2-patch-mtls 2026-03-20 16:26:25.657120 | controller | | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-20 16:26:25.657127 | controller | \ logger.go:42: 16:20:02 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path 2026-03-20 16:26:25.657131 | controller | = /var/lib/watcher transport_url = 'rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.657135 | controller | control_exchange = watcher debug = True '#' empty notification_level means that 2026-03-20 16:26:25.657139 | controller | no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-20 16:26:25.657155 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-20 16:26:25.657160 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.657166 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.657171 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.657175 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.657180 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.657184 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.657190 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.657194 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.657199 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-20 16:26:25.657203 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-20 16:26:25.657207 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-20 16:26:25.657220 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-20 16:26:25.657224 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-20 16:26:25.657228 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.657232 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.657237 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.657243 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]' 2026-03-20 16:26:25.657247 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-20 16:26:25.657252 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.657256 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-20 16:26:25.657260 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-20 16:26:25.657264 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.657268 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-20 16:26:25.657273 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-20 16:26:25.657277 | controller | period = 900\n logger.go:42: 16:20:02 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-20 16:26:25.657281 | controller | 1 ']'\n logger.go:42: 16:20:02 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_keyfile 2026-03-20 16:26:25.657285 | controller | = /etc/pki/tls/private/mtls.key'\n logger.go:42: 16:20:02 | watcher-tls/2-patch-mtls 2026-03-20 16:26:25.657289 | controller | | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-20 16:26:25.657294 | controller | \ logger.go:42: 16:20:02 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path 2026-03-20 16:26:25.657300 | controller | = /var/lib/watcher transport_url = 'rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.657314 | controller | control_exchange = watcher debug = True '#' empty notification_level means that 2026-03-20 16:26:25.657320 | controller | no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-20 16:26:25.657326 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-20 16:26:25.657332 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.657338 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.657343 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.657350 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.657355 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.657359 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.657363 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.657368 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.657372 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-20 16:26:25.657376 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-20 16:26:25.657380 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-20 16:26:25.657385 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-20 16:26:25.657389 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-20 16:26:25.657393 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.657397 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.657401 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.657406 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]' 2026-03-20 16:26:25.657410 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-20 16:26:25.657414 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.657418 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-20 16:26:25.657422 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-20 16:26:25.657429 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.657433 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-20 16:26:25.657438 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-20 16:26:25.657442 | controller | period = 900\n logger.go:42: 16:20:02 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-20 16:26:25.657446 | controller | 1 ']'\n logger.go:42: 16:20:02 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_cafile 2026-03-20 16:26:25.657450 | controller | = /etc/pki/tls/certs/mtls-ca.crt'\n logger.go:42: 16:20:02 | watcher-tls/2-patch-mtls 2026-03-20 16:26:25.657455 | controller | | +++ oc rsh -c watcher-applier watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-20 16:26:25.657459 | controller | \ logger.go:42: 16:20:02 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path 2026-03-20 16:26:25.657463 | controller | = /var/lib/watcher transport_url = 'rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.657467 | controller | control_exchange = watcher debug = True '#' empty notification_level means that 2026-03-20 16:26:25.657473 | controller | no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-20 16:26:25.657478 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-20 16:26:25.657484 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.657492 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.657507 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.657513 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.657519 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.657523 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.657527 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.657531 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.657535 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-20 16:26:25.657540 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-20 16:26:25.657544 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-20 16:26:25.657548 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-20 16:26:25.657561 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-20 16:26:25.657565 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.657569 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.657573 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.657578 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]' 2026-03-20 16:26:25.657582 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-20 16:26:25.657586 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.657593 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-20 16:26:25.657598 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-20 16:26:25.657602 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.657606 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-20 16:26:25.657610 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-20 16:26:25.657614 | controller | period = 900\n logger.go:42: 16:20:02 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-20 16:26:25.657619 | controller | 1 ']'\n logger.go:42: 16:20:02 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_enabled 2026-03-20 16:26:25.657623 | controller | = true'\n logger.go:42: 16:20:02 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier 2026-03-20 16:26:25.657627 | controller | watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 2026-03-20 16:26:25.657631 | controller | 16:20:02 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher 2026-03-20 16:26:25.657635 | controller | transport_url = 'rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.657640 | controller | control_exchange = watcher debug = True '#' empty notification_level means that 2026-03-20 16:26:25.657644 | controller | no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-20 16:26:25.657650 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-20 16:26:25.657654 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.657659 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.657663 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.657667 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.657671 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.657675 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.657680 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.657684 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.657688 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-20 16:26:25.657708 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-20 16:26:25.657724 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-20 16:26:25.657729 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-20 16:26:25.657735 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-20 16:26:25.657740 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.657746 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.657752 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.657756 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]' 2026-03-20 16:26:25.657763 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-20 16:26:25.657767 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.657771 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-20 16:26:25.657775 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-20 16:26:25.657779 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.657784 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-20 16:26:25.657788 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-20 16:26:25.657792 | controller | period = 900\n logger.go:42: 16:20:02 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-20 16:26:25.657796 | controller | 1 ']'\n logger.go:42: 16:20:02 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt\n 2026-03-20 16:26:25.657809 | controller | \ logger.go:42: 16:20:02 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier 2026-03-20 16:26:25.657813 | controller | watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 2026-03-20 16:26:25.657818 | controller | 16:20:03 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher 2026-03-20 16:26:25.657822 | controller | transport_url = 'rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.657826 | controller | control_exchange = watcher debug = True '#' empty notification_level means that 2026-03-20 16:26:25.657833 | controller | no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-20 16:26:25.657838 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-20 16:26:25.657842 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.657846 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.657850 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.657854 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.657859 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.657863 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.657867 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.657871 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.657875 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-20 16:26:25.657879 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-20 16:26:25.657884 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-20 16:26:25.657888 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-20 16:26:25.657892 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-20 16:26:25.657896 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.657901 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.657907 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.657912 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]' 2026-03-20 16:26:25.657926 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-20 16:26:25.657931 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.657936 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-20 16:26:25.657941 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-20 16:26:25.657945 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.657949 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-20 16:26:25.657955 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-20 16:26:25.657960 | controller | period = 900\n logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-20 16:26:25.657966 | controller | 1 ']'\n logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key\n 2026-03-20 16:26:25.657971 | controller | \ logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier 2026-03-20 16:26:25.657975 | controller | watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 2026-03-20 16:26:25.657980 | controller | 16:20:03 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher 2026-03-20 16:26:25.657988 | controller | transport_url = 'rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.657993 | controller | control_exchange = watcher debug = True '#' empty notification_level means that 2026-03-20 16:26:25.657997 | controller | no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-20 16:26:25.658001 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-20 16:26:25.658005 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.658010 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.658014 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.658018 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.658022 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.658026 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.658031 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.658035 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.658039 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-20 16:26:25.658043 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-20 16:26:25.658047 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-20 16:26:25.658053 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-20 16:26:25.658061 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-20 16:26:25.658068 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.658073 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.658079 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.658084 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]' 2026-03-20 16:26:25.658090 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-20 16:26:25.658096 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.658100 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-20 16:26:25.658105 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-20 16:26:25.658109 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.658113 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-20 16:26:25.658128 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-20 16:26:25.658133 | controller | period = 900\n logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-20 16:26:25.658138 | controller | 1 ']'\n logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\n 2026-03-20 16:26:25.658142 | controller | \ logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier 2026-03-20 16:26:25.658149 | controller | watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 2026-03-20 16:26:25.658153 | controller | 16:20:03 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path = /var/lib/watcher 2026-03-20 16:26:25.658158 | controller | transport_url = 'rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.658164 | controller | control_exchange = watcher debug = True '#' empty notification_level means that 2026-03-20 16:26:25.658170 | controller | no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-20 16:26:25.658176 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-20 16:26:25.658181 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.658187 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.658191 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.658197 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.658203 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.658208 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.658214 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.658219 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.658224 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-20 16:26:25.658228 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-20 16:26:25.658232 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-20 16:26:25.658236 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-20 16:26:25.658240 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-20 16:26:25.658245 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.658249 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.658253 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.658257 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[aetos_client]' 2026-03-20 16:26:25.658261 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-20 16:26:25.658266 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.658270 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-20 16:26:25.658274 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-20 16:26:25.658278 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.658282 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-20 16:26:25.658287 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-20 16:26:25.658291 | controller | period = 900\n logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-20 16:26:25.658297 | controller | 1 ']'\n logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | + '[' -n watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.658302 | controller | ']'\n logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | + echo 'Checking watcher-decision-engine 2026-03-20 16:26:25.658306 | controller | config...'\n logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | Checking watcher-decision-engine 2026-03-20 16:26:25.658310 | controller | config...\n logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_certfile 2026-03-20 16:26:25.658314 | controller | = /etc/pki/tls/certs/mtls.crt '\n logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls 2026-03-20 16:26:25.658328 | controller | | +++ oc rsh -c watcher-decision-engine watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-20 16:26:25.658333 | controller | \ logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path 2026-03-20 16:26:25.658337 | controller | = /var/lib/watcher transport_url = 'rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.658341 | controller | control_exchange = watcher debug = True '#' empty notification_level means that 2026-03-20 16:26:25.658346 | controller | no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-20 16:26:25.658350 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-20 16:26:25.658354 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.658358 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.658362 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.658367 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.658371 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.658375 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.658379 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.658383 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.658388 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-20 16:26:25.658394 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-20 16:26:25.658398 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-20 16:26:25.658402 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-20 16:26:25.658407 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-20 16:26:25.658411 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.658415 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.658419 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.658423 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-20 16:26:25.658428 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.658432 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type 2026-03-20 16:26:25.658436 | controller | = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.658442 | controller | = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' 2026-03-20 16:26:25.658447 | controller | endpoint_type = internal region_name = regionOne '[keystone_client]' interface = 2026-03-20 16:26:25.658451 | controller | internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.658455 | controller | = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' 2026-03-20 16:26:25.658460 | controller | interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-20 16:26:25.658464 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-20 16:26:25.658468 | controller | period = 900\n logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-20 16:26:25.658472 | controller | 1 ']'\n logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_keyfile 2026-03-20 16:26:25.658476 | controller | = /etc/pki/tls/private/mtls.key'\n logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls 2026-03-20 16:26:25.658481 | controller | | +++ oc rsh -c watcher-decision-engine watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-20 16:26:25.658486 | controller | \ logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path 2026-03-20 16:26:25.658491 | controller | = /var/lib/watcher transport_url = 'rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.658497 | controller | control_exchange = watcher debug = True '#' empty notification_level means that 2026-03-20 16:26:25.658510 | controller | no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-20 16:26:25.658515 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-20 16:26:25.658520 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.658524 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.658531 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.658536 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.658540 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.658544 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.658548 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.658553 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.658557 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-20 16:26:25.658561 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-20 16:26:25.658565 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-20 16:26:25.658580 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-20 16:26:25.658584 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-20 16:26:25.658588 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.658593 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.658600 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.658606 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-20 16:26:25.658610 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.658614 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type 2026-03-20 16:26:25.658618 | controller | = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.658623 | controller | = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' 2026-03-20 16:26:25.658627 | controller | endpoint_type = internal region_name = regionOne '[keystone_client]' interface = 2026-03-20 16:26:25.658631 | controller | internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.658643 | controller | = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' 2026-03-20 16:26:25.658648 | controller | interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-20 16:26:25.658652 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-20 16:26:25.658656 | controller | period = 900\n logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-20 16:26:25.658660 | controller | 1 ']'\n logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_cafile 2026-03-20 16:26:25.658665 | controller | = /etc/pki/tls/certs/mtls-ca.crt'\n logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls 2026-03-20 16:26:25.658669 | controller | | +++ oc rsh -c watcher-decision-engine watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-20 16:26:25.658673 | controller | \ logger.go:42: 16:20:04 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path 2026-03-20 16:26:25.658677 | controller | = /var/lib/watcher transport_url = 'rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.658683 | controller | control_exchange = watcher debug = True '#' empty notification_level means that 2026-03-20 16:26:25.658689 | controller | no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-20 16:26:25.658712 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-20 16:26:25.658720 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.658731 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.658736 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.658740 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.658744 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.658749 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.658753 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.658757 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.658761 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-20 16:26:25.658765 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-20 16:26:25.658769 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-20 16:26:25.658776 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-20 16:26:25.658780 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-20 16:26:25.658784 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.658788 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.658793 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.658797 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-20 16:26:25.658801 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.658805 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type 2026-03-20 16:26:25.658809 | controller | = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.658813 | controller | = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' 2026-03-20 16:26:25.658818 | controller | endpoint_type = internal region_name = regionOne '[keystone_client]' interface = 2026-03-20 16:26:25.658823 | controller | internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.658828 | controller | = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' 2026-03-20 16:26:25.658834 | controller | interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-20 16:26:25.658839 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-20 16:26:25.658845 | controller | period = 900\n logger.go:42: 16:20:04 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-20 16:26:25.658849 | controller | 1 ']'\n logger.go:42: 16:20:04 | watcher-tls/2-patch-mtls | ++ grep -c 'memcache_tls_enabled 2026-03-20 16:26:25.658854 | controller | = true'\n logger.go:42: 16:20:04 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine 2026-03-20 16:26:25.658858 | controller | watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-20 16:26:25.658862 | controller | \ logger.go:42: 16:20:04 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path 2026-03-20 16:26:25.658866 | controller | = /var/lib/watcher transport_url = 'rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.658870 | controller | control_exchange = watcher debug = True '#' empty notification_level means that 2026-03-20 16:26:25.658875 | controller | no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-20 16:26:25.658881 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-20 16:26:25.658885 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.658890 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.658894 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.658898 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.658902 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.658907 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.658911 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.658929 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.658935 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-20 16:26:25.658941 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-20 16:26:25.658947 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-20 16:26:25.658952 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-20 16:26:25.658958 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-20 16:26:25.658962 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.658966 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.658971 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.658975 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-20 16:26:25.658979 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.658984 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type 2026-03-20 16:26:25.658990 | controller | = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.658999 | controller | = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' 2026-03-20 16:26:25.659007 | controller | endpoint_type = internal region_name = regionOne '[keystone_client]' interface = 2026-03-20 16:26:25.659012 | controller | internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.659018 | controller | = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' 2026-03-20 16:26:25.659023 | controller | interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-20 16:26:25.659029 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-20 16:26:25.659034 | controller | period = 900\n logger.go:42: 16:20:04 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-20 16:26:25.659040 | controller | 1 ']'\n logger.go:42: 16:20:04 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt\n 2026-03-20 16:26:25.659045 | controller | \ logger.go:42: 16:20:04 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine 2026-03-20 16:26:25.659051 | controller | watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-20 16:26:25.659056 | controller | \ logger.go:42: 16:20:04 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path 2026-03-20 16:26:25.659060 | controller | = /var/lib/watcher transport_url = 'rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.659065 | controller | control_exchange = watcher debug = True '#' empty notification_level means that 2026-03-20 16:26:25.659070 | controller | no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-20 16:26:25.659076 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-20 16:26:25.659082 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.659087 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.659093 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.659101 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.659105 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.659109 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.659113 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.659117 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.659121 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-20 16:26:25.659126 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-20 16:26:25.659130 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-20 16:26:25.659134 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-20 16:26:25.659147 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-20 16:26:25.659161 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.659165 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.659170 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.659174 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-20 16:26:25.659178 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.659182 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type 2026-03-20 16:26:25.659186 | controller | = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.659190 | controller | = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' 2026-03-20 16:26:25.659195 | controller | endpoint_type = internal region_name = regionOne '[keystone_client]' interface = 2026-03-20 16:26:25.659199 | controller | internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.659203 | controller | = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' 2026-03-20 16:26:25.659207 | controller | interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-20 16:26:25.659211 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-20 16:26:25.659215 | controller | period = 900\n logger.go:42: 16:20:04 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-20 16:26:25.659219 | controller | 1 ']'\n logger.go:42: 16:20:04 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine 2026-03-20 16:26:25.659224 | controller | watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-20 16:26:25.659228 | controller | \ logger.go:42: 16:20:04 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key\n 2026-03-20 16:26:25.659232 | controller | \ logger.go:42: 16:20:04 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path 2026-03-20 16:26:25.659236 | controller | = /var/lib/watcher transport_url = 'rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.659240 | controller | control_exchange = watcher debug = True '#' empty notification_level means that 2026-03-20 16:26:25.659245 | controller | no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-20 16:26:25.659251 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-20 16:26:25.659255 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.659259 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.659263 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.659268 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.659272 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.659276 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.659280 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.659284 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.659288 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-20 16:26:25.659293 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-20 16:26:25.659297 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-20 16:26:25.659301 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-20 16:26:25.659308 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-20 16:26:25.659314 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.659319 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.659325 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.659330 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-20 16:26:25.659334 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.659346 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type 2026-03-20 16:26:25.659351 | controller | = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.659356 | controller | = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' 2026-03-20 16:26:25.659360 | controller | endpoint_type = internal region_name = regionOne '[keystone_client]' interface = 2026-03-20 16:26:25.659365 | controller | internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.659369 | controller | = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' 2026-03-20 16:26:25.659373 | controller | interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-20 16:26:25.659377 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-20 16:26:25.659381 | controller | period = 900\n logger.go:42: 16:20:04 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-20 16:26:25.659386 | controller | 1 ']'\n logger.go:42: 16:20:04 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt\n 2026-03-20 16:26:25.659390 | controller | \ logger.go:42: 16:20:04 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine 2026-03-20 16:26:25.659394 | controller | watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-20 16:26:25.659398 | controller | \ logger.go:42: 16:20:04 | watcher-tls/2-patch-mtls | ++ echo '[DEFAULT]' state_path 2026-03-20 16:26:25.659404 | controller | = /var/lib/watcher transport_url = 'rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.659409 | controller | control_exchange = watcher debug = True '#' empty notification_level means that 2026-03-20 16:26:25.659415 | controller | no notification will be sent notification_level = '[database]' connection = 'mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-20 16:26:25.659419 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-20 16:26:25.659423 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.659428 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.659436 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.659442 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.659447 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.659453 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.659458 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.659462 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.659466 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-20 16:26:25.659471 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-20 16:26:25.659475 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-20 16:26:25.659479 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-20 16:26:25.659483 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-20 16:26:25.659487 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.659491 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.659496 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.659500 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-20 16:26:25.659504 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.659508 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt '[aetos_client]' endpoint_type 2026-03-20 16:26:25.659512 | controller | = internal region_name = regionOne '[cinder_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.659516 | controller | = regionOne '[glance_client]' endpoint_type = internal region_name = regionOne '[ironic_client]' 2026-03-20 16:26:25.659521 | controller | endpoint_type = internal region_name = regionOne '[keystone_client]' interface = 2026-03-20 16:26:25.659525 | controller | internal region_name = regionOne '[neutron_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.659537 | controller | = regionOne '[nova_client]' endpoint_type = internal region_name = regionOne '[placement_client]' 2026-03-20 16:26:25.659542 | controller | interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-20 16:26:25.659546 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-20 16:26:25.659550 | controller | period = 900\n logger.go:42: 16:20:04 | watcher-tls/2-patch-mtls | + '[' 1 == 2026-03-20 16:26:25.659626 | controller | 1 ']'\n logger.go:42: 16:20:04 | watcher-tls/2-patch-mtls | test step completed 2026-03-20 16:26:25.659631 | controller | 2-patch-mtls\n logger.go:42: 16:20:04 | watcher-tls/3-disable-podlevel-tls | 2026-03-20 16:26:25.659636 | controller | starting test step 3-disable-podlevel-tls\n logger.go:42: 16:20:04 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.659641 | controller | | running command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type='json' 2026-03-20 16:26:25.659646 | controller | -p='[{\"op\": \"replace\", \"path\": \"/spec/apiServiceTemplate/tls/api\", \"value\":{ 2026-03-20 16:26:25.659651 | controller | \"internal\": {}, \"public\": {} }}]'\n ]\n logger.go:42: 16:20:05 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.659656 | controller | | watcher.watcher.openstack.org/watcher-kuttl patched\n logger.go:42: 16:20:05 2026-03-20 16:26:25.659663 | controller | | watcher-tls/3-disable-podlevel-tls | running command: [sh -c set -euxo pipefail\n 2026-03-20 16:26:25.659668 | controller | \ oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-20 16:26:25.659673 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc exec -n ${NAMESPACE} 2026-03-20 16:26:25.659678 | controller | openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep 2026-03-20 16:26:25.659683 | controller | watcher| awk '{print $1}')\n [ $(oc get -n ${NAMESPACE} keystoneservice watcher 2026-03-20 16:26:25.659688 | controller | -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ -n \"$(oc get -n ${NAMESPACE} 2026-03-20 16:26:25.659706 | controller | watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n # check that 2026-03-20 16:26:25.659714 | controller | watcher internal endpoint does not use https\n oc exec -n ${NAMESPACE} openstackclient 2026-03-20 16:26:25.659719 | controller | -- openstack endpoint list | grep infra-optim | grep internal | [ $(grep -c https) 2026-03-20 16:26:25.659724 | controller | == 0 ]\n # check that watcher public endpoint does use https\n oc 2026-03-20 16:26:25.659728 | controller | exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim 2026-03-20 16:26:25.659733 | controller | | grep public | [ $(grep -c https) == 1 ]\n # If we are running the container 2026-03-20 16:26:25.659738 | controller | locally, skip following test\n if [ \"$(oc get pods -n openstack-operators 2026-03-20 16:26:25.659743 | controller | -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 2026-03-20 16:26:25.659748 | controller | 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators 2026-03-20 16:26:25.659753 | controller | -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n 2026-03-20 16:26:25.659758 | controller | \ counter=0\n for i in ${env_variables}; do\n if echo ${i} 2026-03-20 16:26:25.659763 | controller | | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter 2026-03-20 16:26:25.659767 | controller | + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo 2026-03-20 16:26:25.659772 | controller | \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n 2026-03-20 16:26:25.659777 | controller | \ echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n 2026-03-20 16:26:25.659782 | controller | \ ]\n logger.go:42: 16:20:05 | watcher-tls/3-disable-podlevel-tls | + oc 2026-03-20 16:26:25.659787 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-20 16:26:25.659792 | controller | -c Name -c Type\n logger.go:42: 16:20:05 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.659797 | controller | | ++ grep -c '^watcher'\n logger.go:42: 16:20:07 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.659802 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:20:07 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.659807 | controller | | ++ grep watcher\n logger.go:42: 16:20:07 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.659813 | controller | | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.659818 | controller | -f value -c Name -c Type -c ID\n logger.go:42: 16:20:07 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.659822 | controller | | ++ awk '{print $1}'\n logger.go:42: 16:20:10 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.659826 | controller | | + SERVICEID=0bc8267151fd4fefbc0a3327396365a4\n logger.go:42: 16:20:10 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.659830 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-20 16:26:25.659842 | controller | \ logger.go:42: 16:20:10 | watcher-tls/3-disable-podlevel-tls | + '[' 0bc8267151fd4fefbc0a3327396365a4 2026-03-20 16:26:25.659848 | controller | == 0bc8267151fd4fefbc0a3327396365a4 ']'\n logger.go:42: 16:20:10 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.659852 | controller | | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n 2026-03-20 16:26:25.659857 | controller | \ logger.go:42: 16:20:10 | watcher-tls/3-disable-podlevel-tls | + '[' -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.659865 | controller | ']'\n logger.go:42: 16:20:10 | watcher-tls/3-disable-podlevel-tls | + grep internal\n 2026-03-20 16:26:25.659871 | controller | \ logger.go:42: 16:20:10 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim\n 2026-03-20 16:26:25.659877 | controller | \ logger.go:42: 16:20:10 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https\n 2026-03-20 16:26:25.659882 | controller | \ logger.go:42: 16:20:10 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.659886 | controller | openstackclient -- openstack endpoint list\n logger.go:42: 16:20:12 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.659890 | controller | | + '[' 1 == 0 ']'\n logger.go:42: 16:20:13 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.659895 | controller | | running command: [sh -c set -euxo pipefail\n oc exec -n ${NAMESPACE} openstackclient 2026-03-20 16:26:25.659899 | controller | -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n 2026-03-20 16:26:25.659903 | controller | \ SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-20 16:26:25.659907 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ 2026-03-20 16:26:25.659913 | controller | $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.659918 | controller | == ${SERVICEID} ]\n [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl 2026-03-20 16:26:25.659922 | controller | -o jsonpath={.status.hash.dbsync})\" ]\n # check that watcher internal endpoint 2026-03-20 16:26:25.659926 | controller | does not use https\n oc exec -n ${NAMESPACE} openstackclient -- openstack 2026-03-20 16:26:25.659930 | controller | endpoint list | grep infra-optim | grep internal | [ $(grep -c https) == 0 ]\n 2026-03-20 16:26:25.659934 | controller | \ # check that watcher public endpoint does use https\n oc exec -n 2026-03-20 16:26:25.659939 | controller | ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | grep 2026-03-20 16:26:25.659943 | controller | public | [ $(grep -c https) == 1 ]\n # If we are running the container locally, 2026-03-20 16:26:25.659947 | controller | skip following test\n if [ \"$(oc get pods -n openstack-operators -o name 2026-03-20 16:26:25.659951 | controller | -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 0\n 2026-03-20 16:26:25.659955 | controller | \ fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators 2026-03-20 16:26:25.659960 | controller | -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n 2026-03-20 16:26:25.659964 | controller | \ counter=0\n for i in ${env_variables}; do\n if echo ${i} 2026-03-20 16:26:25.659972 | controller | | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter 2026-03-20 16:26:25.659978 | controller | + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo 2026-03-20 16:26:25.659983 | controller | \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n 2026-03-20 16:26:25.659989 | controller | \ echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n 2026-03-20 16:26:25.659995 | controller | \ ]\n logger.go:42: 16:20:13 | watcher-tls/3-disable-podlevel-tls | + oc 2026-03-20 16:26:25.660000 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-20 16:26:25.660005 | controller | -c Name -c Type\n logger.go:42: 16:20:13 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660009 | controller | | ++ grep -c '^watcher'\n logger.go:42: 16:20:15 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660013 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:20:15 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660017 | controller | | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.660021 | controller | -f value -c Name -c Type -c ID\n logger.go:42: 16:20:15 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660025 | controller | | ++ awk '{print $1}'\n logger.go:42: 16:20:15 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660029 | controller | | ++ grep watcher\n logger.go:42: 16:20:17 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660034 | controller | | + SERVICEID=0bc8267151fd4fefbc0a3327396365a4\n logger.go:42: 16:20:17 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660038 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-20 16:26:25.660051 | controller | \ logger.go:42: 16:20:18 | watcher-tls/3-disable-podlevel-tls | + '[' 0bc8267151fd4fefbc0a3327396365a4 2026-03-20 16:26:25.660055 | controller | == 0bc8267151fd4fefbc0a3327396365a4 ']'\n logger.go:42: 16:20:18 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660060 | controller | | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n 2026-03-20 16:26:25.660064 | controller | \ logger.go:42: 16:20:18 | watcher-tls/3-disable-podlevel-tls | + '[' -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.660068 | controller | ']'\n logger.go:42: 16:20:18 | watcher-tls/3-disable-podlevel-tls | + grep internal\n 2026-03-20 16:26:25.660072 | controller | \ logger.go:42: 16:20:18 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim\n 2026-03-20 16:26:25.660077 | controller | \ logger.go:42: 16:20:18 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.660091 | controller | openstackclient -- openstack endpoint list\n logger.go:42: 16:20:18 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660095 | controller | | ++ grep -c https\n logger.go:42: 16:20:20 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660099 | controller | | + '[' 1 == 0 ']'\n logger.go:42: 16:20:21 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660106 | controller | | running command: [sh -c set -euxo pipefail\n oc exec -n ${NAMESPACE} openstackclient 2026-03-20 16:26:25.660110 | controller | -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n 2026-03-20 16:26:25.660114 | controller | \ SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-20 16:26:25.660118 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ 2026-03-20 16:26:25.660122 | controller | $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.660129 | controller | == ${SERVICEID} ]\n [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl 2026-03-20 16:26:25.660134 | controller | -o jsonpath={.status.hash.dbsync})\" ]\n # check that watcher internal endpoint 2026-03-20 16:26:25.660141 | controller | does not use https\n oc exec -n ${NAMESPACE} openstackclient -- openstack 2026-03-20 16:26:25.660147 | controller | endpoint list | grep infra-optim | grep internal | [ $(grep -c https) == 0 ]\n 2026-03-20 16:26:25.660152 | controller | \ # check that watcher public endpoint does use https\n oc exec -n 2026-03-20 16:26:25.660158 | controller | ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | grep 2026-03-20 16:26:25.660163 | controller | public | [ $(grep -c https) == 1 ]\n # If we are running the container locally, 2026-03-20 16:26:25.660168 | controller | skip following test\n if [ \"$(oc get pods -n openstack-operators -o name 2026-03-20 16:26:25.660180 | controller | -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 0\n 2026-03-20 16:26:25.660185 | controller | \ fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators 2026-03-20 16:26:25.660189 | controller | -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n 2026-03-20 16:26:25.660193 | controller | \ counter=0\n for i in ${env_variables}; do\n if echo ${i} 2026-03-20 16:26:25.660197 | controller | | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter 2026-03-20 16:26:25.660201 | controller | + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo 2026-03-20 16:26:25.660205 | controller | \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n 2026-03-20 16:26:25.660210 | controller | \ echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n 2026-03-20 16:26:25.660214 | controller | \ ]\n logger.go:42: 16:20:21 | watcher-tls/3-disable-podlevel-tls | + oc 2026-03-20 16:26:25.660218 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-20 16:26:25.660222 | controller | -c Name -c Type\n logger.go:42: 16:20:21 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660226 | controller | | ++ grep -c '^watcher'\n logger.go:42: 16:20:23 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660230 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:20:23 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660235 | controller | | ++ grep watcher\n logger.go:42: 16:20:23 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660239 | controller | | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.660243 | controller | -f value -c Name -c Type -c ID\n logger.go:42: 16:20:23 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660247 | controller | | ++ awk '{print $1}'\n logger.go:42: 16:20:25 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660259 | controller | | + SERVICEID=0bc8267151fd4fefbc0a3327396365a4\n logger.go:42: 16:20:25 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660263 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-20 16:26:25.660268 | controller | \ logger.go:42: 16:20:25 | watcher-tls/3-disable-podlevel-tls | + '[' 0bc8267151fd4fefbc0a3327396365a4 2026-03-20 16:26:25.660279 | controller | == 0bc8267151fd4fefbc0a3327396365a4 ']'\n logger.go:42: 16:20:25 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660284 | controller | | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n 2026-03-20 16:26:25.660288 | controller | \ logger.go:42: 16:20:25 | watcher-tls/3-disable-podlevel-tls | + '[' -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.660292 | controller | ']'\n logger.go:42: 16:20:25 | watcher-tls/3-disable-podlevel-tls | + grep internal\n 2026-03-20 16:26:25.660297 | controller | \ logger.go:42: 16:20:25 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https\n 2026-03-20 16:26:25.660301 | controller | \ logger.go:42: 16:20:25 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim\n 2026-03-20 16:26:25.660305 | controller | \ logger.go:42: 16:20:25 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.660312 | controller | openstackclient -- openstack endpoint list\n logger.go:42: 16:20:27 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660316 | controller | | + '[' 1 == 0 ']'\n logger.go:42: 16:20:28 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660321 | controller | | running command: [sh -c set -euxo pipefail\n oc exec -n ${NAMESPACE} openstackclient 2026-03-20 16:26:25.660325 | controller | -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n 2026-03-20 16:26:25.660329 | controller | \ SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-20 16:26:25.660333 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ 2026-03-20 16:26:25.660337 | controller | $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.660341 | controller | == ${SERVICEID} ]\n [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl 2026-03-20 16:26:25.660349 | controller | -o jsonpath={.status.hash.dbsync})\" ]\n # check that watcher internal endpoint 2026-03-20 16:26:25.660355 | controller | does not use https\n oc exec -n ${NAMESPACE} openstackclient -- openstack 2026-03-20 16:26:25.660360 | controller | endpoint list | grep infra-optim | grep internal | [ $(grep -c https) == 0 ]\n 2026-03-20 16:26:25.660366 | controller | \ # check that watcher public endpoint does use https\n oc exec -n 2026-03-20 16:26:25.660371 | controller | ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | grep 2026-03-20 16:26:25.660377 | controller | public | [ $(grep -c https) == 1 ]\n # If we are running the container locally, 2026-03-20 16:26:25.660382 | controller | skip following test\n if [ \"$(oc get pods -n openstack-operators -o name 2026-03-20 16:26:25.660387 | controller | -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 0\n 2026-03-20 16:26:25.660391 | controller | \ fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators 2026-03-20 16:26:25.660395 | controller | -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n 2026-03-20 16:26:25.660399 | controller | \ counter=0\n for i in ${env_variables}; do\n if echo ${i} 2026-03-20 16:26:25.660404 | controller | | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter 2026-03-20 16:26:25.660408 | controller | + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo 2026-03-20 16:26:25.660412 | controller | \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n 2026-03-20 16:26:25.660416 | controller | \ echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n 2026-03-20 16:26:25.660420 | controller | \ ]\n logger.go:42: 16:20:28 | watcher-tls/3-disable-podlevel-tls | + oc 2026-03-20 16:26:25.660424 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-20 16:26:25.660429 | controller | -c Name -c Type\n logger.go:42: 16:20:28 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660433 | controller | | ++ grep -c '^watcher'\n logger.go:42: 16:20:30 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660437 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:20:30 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660441 | controller | | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.660445 | controller | -f value -c Name -c Type -c ID\n logger.go:42: 16:20:30 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660449 | controller | | ++ awk '{print $1}'\n logger.go:42: 16:20:30 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660454 | controller | | ++ grep watcher\n logger.go:42: 16:20:32 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660458 | controller | | + SERVICEID=0bc8267151fd4fefbc0a3327396365a4\n logger.go:42: 16:20:32 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660465 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-20 16:26:25.660470 | controller | \ logger.go:42: 16:20:32 | watcher-tls/3-disable-podlevel-tls | + '[' 0bc8267151fd4fefbc0a3327396365a4 2026-03-20 16:26:25.660486 | controller | == 0bc8267151fd4fefbc0a3327396365a4 ']'\n logger.go:42: 16:20:32 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660492 | controller | | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n 2026-03-20 16:26:25.660498 | controller | \ logger.go:42: 16:20:33 | watcher-tls/3-disable-podlevel-tls | + '[' -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.660502 | controller | ']'\n logger.go:42: 16:20:33 | watcher-tls/3-disable-podlevel-tls | + grep internal\n 2026-03-20 16:26:25.660506 | controller | \ logger.go:42: 16:20:33 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim\n 2026-03-20 16:26:25.660511 | controller | \ logger.go:42: 16:20:33 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.660515 | controller | openstackclient -- openstack endpoint list\n logger.go:42: 16:20:33 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660519 | controller | | ++ grep -c https\n logger.go:42: 16:20:34 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660523 | controller | | + '[' 1 == 0 ']'\n logger.go:42: 16:20:36 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660527 | controller | | running command: [sh -c set -euxo pipefail\n oc exec -n ${NAMESPACE} openstackclient 2026-03-20 16:26:25.660531 | controller | -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n 2026-03-20 16:26:25.660536 | controller | \ SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-20 16:26:25.660540 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ 2026-03-20 16:26:25.660544 | controller | $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.660548 | controller | == ${SERVICEID} ]\n [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl 2026-03-20 16:26:25.660552 | controller | -o jsonpath={.status.hash.dbsync})\" ]\n # check that watcher internal endpoint 2026-03-20 16:26:25.660559 | controller | does not use https\n oc exec -n ${NAMESPACE} openstackclient -- openstack 2026-03-20 16:26:25.660564 | controller | endpoint list | grep infra-optim | grep internal | [ $(grep -c https) == 0 ]\n 2026-03-20 16:26:25.660568 | controller | \ # check that watcher public endpoint does use https\n oc exec -n 2026-03-20 16:26:25.660572 | controller | ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim | grep 2026-03-20 16:26:25.660576 | controller | public | [ $(grep -c https) == 1 ]\n # If we are running the container locally, 2026-03-20 16:26:25.660580 | controller | skip following test\n if [ \"$(oc get pods -n openstack-operators -o name 2026-03-20 16:26:25.660585 | controller | -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 0\n 2026-03-20 16:26:25.660589 | controller | \ fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators 2026-03-20 16:26:25.660593 | controller | -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n 2026-03-20 16:26:25.660597 | controller | \ counter=0\n for i in ${env_variables}; do\n if echo ${i} 2026-03-20 16:26:25.660601 | controller | | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter 2026-03-20 16:26:25.660606 | controller | + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo 2026-03-20 16:26:25.660610 | controller | \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n 2026-03-20 16:26:25.660614 | controller | \ echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n 2026-03-20 16:26:25.660618 | controller | \ ]\n logger.go:42: 16:20:36 | watcher-tls/3-disable-podlevel-tls | + oc 2026-03-20 16:26:25.660624 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-20 16:26:25.660637 | controller | -c Name -c Type\n logger.go:42: 16:20:36 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660642 | controller | | ++ grep -c '^watcher'\n logger.go:42: 16:20:38 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660646 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:20:38 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660650 | controller | | ++ grep watcher\n logger.go:42: 16:20:38 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660654 | controller | | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.660658 | controller | -f value -c Name -c Type -c ID\n logger.go:42: 16:20:38 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660663 | controller | | ++ awk '{print $1}'\n logger.go:42: 16:20:40 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660667 | controller | | + SERVICEID=0bc8267151fd4fefbc0a3327396365a4\n logger.go:42: 16:20:40 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660671 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-20 16:26:25.660675 | controller | \ logger.go:42: 16:20:40 | watcher-tls/3-disable-podlevel-tls | + '[' 0bc8267151fd4fefbc0a3327396365a4 2026-03-20 16:26:25.660679 | controller | == 0bc8267151fd4fefbc0a3327396365a4 ']'\n logger.go:42: 16:20:40 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660703 | controller | | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n 2026-03-20 16:26:25.660711 | controller | \ logger.go:42: 16:20:40 | watcher-tls/3-disable-podlevel-tls | + '[' -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.660716 | controller | ']'\n logger.go:42: 16:20:40 | watcher-tls/3-disable-podlevel-tls | + grep internal\n 2026-03-20 16:26:25.660720 | controller | \ logger.go:42: 16:20:40 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim\n 2026-03-20 16:26:25.660724 | controller | \ logger.go:42: 16:20:40 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.660728 | controller | openstackclient -- openstack endpoint list\n logger.go:42: 16:20:40 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660732 | controller | | ++ grep -c https\n logger.go:42: 16:20:47 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660739 | controller | | + '[' 0 == 0 ']'\n logger.go:42: 16:20:47 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660743 | controller | | + grep public\n logger.go:42: 16:20:47 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660747 | controller | | + grep infra-optim\n logger.go:42: 16:20:47 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660752 | controller | | ++ grep -c https\n logger.go:42: 16:20:47 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660756 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack endpoint list\n 2026-03-20 16:26:25.660760 | controller | \ logger.go:42: 16:20:49 | watcher-tls/3-disable-podlevel-tls | + '[' 1 == 1 ']'\n 2026-03-20 16:26:25.660764 | controller | \ logger.go:42: 16:20:49 | watcher-tls/3-disable-podlevel-tls | ++ oc get pods 2026-03-20 16:26:25.660768 | controller | -n openstack-operators -o name -l openstack.org/operator-name=watcher\n logger.go:42: 2026-03-20 16:26:25.660772 | controller | 16:20:49 | watcher-tls/3-disable-podlevel-tls | + '[' '' == '' ']'\n logger.go:42: 2026-03-20 16:26:25.660778 | controller | 16:20:49 | watcher-tls/3-disable-podlevel-tls | + exit 0\n logger.go:42: 16:20:51 2026-03-20 16:26:25.660782 | controller | | watcher-tls/3-disable-podlevel-tls | running command: [sh -c set -euxo pipefail\n 2026-03-20 16:26:25.660786 | controller | \ oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-20 16:26:25.660791 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc exec -n ${NAMESPACE} 2026-03-20 16:26:25.660797 | controller | openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep 2026-03-20 16:26:25.660801 | controller | watcher| awk '{print $1}')\n [ $(oc get -n ${NAMESPACE} keystoneservice watcher 2026-03-20 16:26:25.660805 | controller | -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ -n \"$(oc get -n ${NAMESPACE} 2026-03-20 16:26:25.660811 | controller | watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n # check that 2026-03-20 16:26:25.660815 | controller | watcher internal endpoint does not use https\n oc exec -n ${NAMESPACE} openstackclient 2026-03-20 16:26:25.660819 | controller | -- openstack endpoint list | grep infra-optim | grep internal | [ $(grep -c https) 2026-03-20 16:26:25.660824 | controller | == 0 ]\n # check that watcher public endpoint does use https\n oc 2026-03-20 16:26:25.660828 | controller | exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep infra-optim 2026-03-20 16:26:25.660832 | controller | | grep public | [ $(grep -c https) == 1 ]\n # If we are running the container 2026-03-20 16:26:25.660836 | controller | locally, skip following test\n if [ \"$(oc get pods -n openstack-operators 2026-03-20 16:26:25.660840 | controller | -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 2026-03-20 16:26:25.660844 | controller | 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators 2026-03-20 16:26:25.660849 | controller | -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n 2026-03-20 16:26:25.660853 | controller | \ counter=0\n for i in ${env_variables}; do\n if echo ${i} 2026-03-20 16:26:25.660857 | controller | | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter 2026-03-20 16:26:25.660861 | controller | + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo 2026-03-20 16:26:25.660865 | controller | \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n 2026-03-20 16:26:25.660869 | controller | \ echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n 2026-03-20 16:26:25.660874 | controller | \ ]\n logger.go:42: 16:20:51 | watcher-tls/3-disable-podlevel-tls | + oc 2026-03-20 16:26:25.660878 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-20 16:26:25.660882 | controller | -c Name -c Type\n logger.go:42: 16:20:51 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660886 | controller | | ++ grep -c '^watcher'\n logger.go:42: 16:20:53 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660890 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:20:53 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660903 | controller | | ++ grep watcher\n logger.go:42: 16:20:53 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660907 | controller | | ++ awk '{print $1}'\n logger.go:42: 16:20:53 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660911 | controller | | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.660916 | controller | -f value -c Name -c Type -c ID\n logger.go:42: 16:20:55 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660920 | controller | | + SERVICEID=0bc8267151fd4fefbc0a3327396365a4\n logger.go:42: 16:20:55 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660924 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-20 16:26:25.660928 | controller | \ logger.go:42: 16:20:55 | watcher-tls/3-disable-podlevel-tls | + '[' 0bc8267151fd4fefbc0a3327396365a4 2026-03-20 16:26:25.660933 | controller | == 0bc8267151fd4fefbc0a3327396365a4 ']'\n logger.go:42: 16:20:55 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660937 | controller | | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n 2026-03-20 16:26:25.660941 | controller | \ logger.go:42: 16:20:55 | watcher-tls/3-disable-podlevel-tls | + '[' -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.660947 | controller | ']'\n logger.go:42: 16:20:55 | watcher-tls/3-disable-podlevel-tls | + grep internal\n 2026-03-20 16:26:25.660951 | controller | \ logger.go:42: 16:20:55 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim\n 2026-03-20 16:26:25.660955 | controller | \ logger.go:42: 16:20:55 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https\n 2026-03-20 16:26:25.660960 | controller | \ logger.go:42: 16:20:55 | watcher-tls/3-disable-podlevel-tls | + oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.660964 | controller | openstackclient -- openstack endpoint list\n logger.go:42: 16:20:57 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660968 | controller | | + '[' 0 == 0 ']'\n logger.go:42: 16:20:57 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660972 | controller | | + grep public\n logger.go:42: 16:20:57 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660978 | controller | | ++ grep -c https\n logger.go:42: 16:20:57 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660982 | controller | | + grep infra-optim\n logger.go:42: 16:20:57 | watcher-tls/3-disable-podlevel-tls 2026-03-20 16:26:25.660986 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack endpoint list\n 2026-03-20 16:26:25.660991 | controller | \ logger.go:42: 16:20:59 | watcher-tls/3-disable-podlevel-tls | + '[' 1 == 1 ']'\n 2026-03-20 16:26:25.660995 | controller | \ logger.go:42: 16:20:59 | watcher-tls/3-disable-podlevel-tls | ++ oc get pods 2026-03-20 16:26:25.660999 | controller | -n openstack-operators -o name -l openstack.org/operator-name=watcher\n logger.go:42: 2026-03-20 16:26:25.661003 | controller | 16:21:00 | watcher-tls/3-disable-podlevel-tls | + '[' '' == '' ']'\n logger.go:42: 2026-03-20 16:26:25.661007 | controller | 16:21:00 | watcher-tls/3-disable-podlevel-tls | + exit 0\n logger.go:42: 16:21:00 2026-03-20 16:26:25.661011 | controller | | watcher-tls/3-disable-podlevel-tls | test step completed 3-disable-podlevel-tls\n 2026-03-20 16:26:25.661015 | controller | \ logger.go:42: 16:21:00 | watcher-tls/4-deploy-without-route | starting test 2026-03-20 16:26:25.661020 | controller | step 4-deploy-without-route\n logger.go:42: 16:21:00 | watcher-tls/4-deploy-without-route 2026-03-20 16:26:25.661024 | controller | | running command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type='json' 2026-03-20 16:26:25.661028 | controller | -p='[{\"op\": \"replace\", \"path\": \"/spec/apiServiceTemplate/override\", \"value\":{\"service\": 2026-03-20 16:26:25.661032 | controller | { \"internal\": {}, \"public\": { \"metadata\": { \"annotations\": { \"metallb.universe.tf/address-pool\": 2026-03-20 16:26:25.661036 | controller | \"ctlplane\", \"metallb.universe.tf/allow-shared-ip\": \"ctlplane\" } }, \"spec\": 2026-03-20 16:26:25.661041 | controller | { \"type\": \"LoadBalancer\" } } } }}]'\n ]\n logger.go:42: 16:21:00 | 2026-03-20 16:26:25.661045 | controller | watcher-tls/4-deploy-without-route | watcher.watcher.openstack.org/watcher-kuttl 2026-03-20 16:26:25.661049 | controller | patched\n logger.go:42: 16:21:00 | watcher-tls/4-deploy-without-route | running 2026-03-20 16:26:25.661053 | controller | command: [sh -c set -euxo pipefail\n oc exec -n ${NAMESPACE} openstackclient 2026-03-20 16:26:25.661057 | controller | -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n 2026-03-20 16:26:25.661061 | controller | \ SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-20 16:26:25.661066 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ 2026-03-20 16:26:25.661070 | controller | $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.661074 | controller | == ${SERVICEID} ]\n [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl 2026-03-20 16:26:25.661078 | controller | -o jsonpath={.status.hash.dbsync})\" ]\n # If we are running the container 2026-03-20 16:26:25.661089 | controller | locally, skip following test\n if [ \"$(oc get pods -n openstack-operators 2026-03-20 16:26:25.661096 | controller | -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 2026-03-20 16:26:25.661100 | controller | 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators 2026-03-20 16:26:25.661104 | controller | -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n 2026-03-20 16:26:25.661108 | controller | \ counter=0\n for i in ${env_variables}; do\n if echo ${i} 2026-03-20 16:26:25.661112 | controller | | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter 2026-03-20 16:26:25.661116 | controller | + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo 2026-03-20 16:26:25.661121 | controller | \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n 2026-03-20 16:26:25.661125 | controller | \ echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n 2026-03-20 16:26:25.661129 | controller | \ ]\n logger.go:42: 16:21:00 | watcher-tls/4-deploy-without-route | + oc 2026-03-20 16:26:25.661133 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-20 16:26:25.661147 | controller | -c Name -c Type\n logger.go:42: 16:21:00 | watcher-tls/4-deploy-without-route 2026-03-20 16:26:25.661151 | controller | | ++ grep -c '^watcher'\n logger.go:42: 16:21:02 | watcher-tls/4-deploy-without-route 2026-03-20 16:26:25.661156 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:21:02 | watcher-tls/4-deploy-without-route 2026-03-20 16:26:25.661160 | controller | | ++ grep watcher\n logger.go:42: 16:21:02 | watcher-tls/4-deploy-without-route 2026-03-20 16:26:25.661164 | controller | | ++ awk '{print $1}'\n logger.go:42: 16:21:02 | watcher-tls/4-deploy-without-route 2026-03-20 16:26:25.661168 | controller | | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.661172 | controller | -f value -c Name -c Type -c ID\n logger.go:42: 16:21:04 | watcher-tls/4-deploy-without-route 2026-03-20 16:26:25.661177 | controller | | + SERVICEID=0bc8267151fd4fefbc0a3327396365a4\n logger.go:42: 16:21:04 | watcher-tls/4-deploy-without-route 2026-03-20 16:26:25.661181 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-20 16:26:25.661185 | controller | \ logger.go:42: 16:21:04 | watcher-tls/4-deploy-without-route | + '[' 0bc8267151fd4fefbc0a3327396365a4 2026-03-20 16:26:25.661189 | controller | == 0bc8267151fd4fefbc0a3327396365a4 ']'\n logger.go:42: 16:21:04 | watcher-tls/4-deploy-without-route 2026-03-20 16:26:25.661193 | controller | | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n 2026-03-20 16:26:25.661197 | controller | \ logger.go:42: 16:21:04 | watcher-tls/4-deploy-without-route | + '[' -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.661202 | controller | ']'\n logger.go:42: 16:21:04 | watcher-tls/4-deploy-without-route | ++ oc get 2026-03-20 16:26:25.661206 | controller | pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n logger.go:42: 2026-03-20 16:26:25.661210 | controller | 16:21:05 | watcher-tls/4-deploy-without-route | + '[' '' == '' ']'\n logger.go:42: 2026-03-20 16:26:25.661214 | controller | 16:21:05 | watcher-tls/4-deploy-without-route | + exit 0\n logger.go:42: 16:21:06 2026-03-20 16:26:25.661218 | controller | | watcher-tls/4-deploy-without-route | running command: [sh -c set -euxo pipefail\n 2026-03-20 16:26:25.661222 | controller | \ oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-20 16:26:25.661226 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc exec -n ${NAMESPACE} 2026-03-20 16:26:25.661231 | controller | openstackclient -- openstack service list -f value -c Name -c Type -c ID | grep 2026-03-20 16:26:25.661235 | controller | watcher| awk '{print $1}')\n [ $(oc get -n ${NAMESPACE} keystoneservice watcher 2026-03-20 16:26:25.661242 | controller | -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ -n \"$(oc get -n ${NAMESPACE} 2026-03-20 16:26:25.661247 | controller | watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" ]\n # If we are 2026-03-20 16:26:25.661251 | controller | running the container locally, skip following test\n if [ \"$(oc get pods 2026-03-20 16:26:25.661255 | controller | -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" == \"\" 2026-03-20 16:26:25.661259 | controller | ]; then\n exit 0\n fi\n env_variables=$(oc set env $(oc 2026-03-20 16:26:25.661263 | controller | get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) 2026-03-20 16:26:25.661267 | controller | -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; 2026-03-20 16:26:25.661272 | controller | do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo 2026-03-20 16:26:25.661276 | controller | ${i}\n counter=$((counter + 1))\n fi\n done\n if 2026-03-20 16:26:25.661280 | controller | [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables 2026-03-20 16:26:25.661292 | controller | found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT 2026-03-20 16:26:25.661297 | controller | variables found.\"\n fi\n ]\n logger.go:42: 16:21:06 | watcher-tls/4-deploy-without-route 2026-03-20 16:26:25.661301 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-20 16:26:25.661305 | controller | value -c Name -c Type\n logger.go:42: 16:21:06 | watcher-tls/4-deploy-without-route 2026-03-20 16:26:25.661309 | controller | | ++ grep -c '^watcher'\n logger.go:42: 16:21:08 | watcher-tls/4-deploy-without-route 2026-03-20 16:26:25.661313 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:21:08 | watcher-tls/4-deploy-without-route 2026-03-20 16:26:25.661317 | controller | | ++ awk '{print $1}'\n logger.go:42: 16:21:08 | watcher-tls/4-deploy-without-route 2026-03-20 16:26:25.661322 | controller | | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.661326 | controller | -f value -c Name -c Type -c ID\n logger.go:42: 16:21:08 | watcher-tls/4-deploy-without-route 2026-03-20 16:26:25.661330 | controller | | ++ grep watcher\n logger.go:42: 16:21:10 | watcher-tls/4-deploy-without-route 2026-03-20 16:26:25.661334 | controller | | + SERVICEID=0bc8267151fd4fefbc0a3327396365a4\n logger.go:42: 16:21:10 | watcher-tls/4-deploy-without-route 2026-03-20 16:26:25.661338 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-20 16:26:25.661342 | controller | \ logger.go:42: 16:21:10 | watcher-tls/4-deploy-without-route | + '[' 0bc8267151fd4fefbc0a3327396365a4 2026-03-20 16:26:25.661347 | controller | == 0bc8267151fd4fefbc0a3327396365a4 ']'\n logger.go:42: 16:21:10 | watcher-tls/4-deploy-without-route 2026-03-20 16:26:25.661353 | controller | | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n 2026-03-20 16:26:25.661357 | controller | \ logger.go:42: 16:21:10 | watcher-tls/4-deploy-without-route | + '[' -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.661362 | controller | ']'\n logger.go:42: 16:21:10 | watcher-tls/4-deploy-without-route | ++ oc get 2026-03-20 16:26:25.661366 | controller | pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n logger.go:42: 2026-03-20 16:26:25.661371 | controller | 16:21:10 | watcher-tls/4-deploy-without-route | + '[' '' == '' ']'\n logger.go:42: 2026-03-20 16:26:25.661375 | controller | 16:21:10 | watcher-tls/4-deploy-without-route | + exit 0\n logger.go:42: 16:21:10 2026-03-20 16:26:25.661381 | controller | | watcher-tls/4-deploy-without-route | test step completed 4-deploy-without-route\n 2026-03-20 16:26:25.661385 | controller | \ logger.go:42: 16:21:10 | watcher-tls/5-disable-tls | starting test step 5-disable-tls\n 2026-03-20 16:26:25.661391 | controller | \ logger.go:42: 16:21:10 | watcher-tls/5-disable-tls | running command: [sh -c 2026-03-20 16:26:25.661395 | controller | oc patch watcher -n $NAMESPACE watcher-kuttl --type='json' -p='[{\"op\": \"replace\", 2026-03-20 16:26:25.661399 | controller | \"path\": \"/spec/apiServiceTemplate/override\", \"value\":{}}]'\n ]\n logger.go:42: 2026-03-20 16:26:25.661403 | controller | 16:21:10 | watcher-tls/5-disable-tls | watcher.watcher.openstack.org/watcher-kuttl 2026-03-20 16:26:25.661407 | controller | patched\n logger.go:42: 16:21:10 | watcher-tls/5-disable-tls | running command: 2026-03-20 16:26:25.661412 | controller | [sh -c set -euxo pipefail\n oc exec -n ${NAMESPACE} openstackclient -- openstack 2026-03-20 16:26:25.661416 | controller | service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc 2026-03-20 16:26:25.661420 | controller | exec -n ${NAMESPACE} openstackclient -- openstack service list -f value -c Name 2026-03-20 16:26:25.661424 | controller | -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n ${NAMESPACE} 2026-03-20 16:26:25.661428 | controller | keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ 2026-03-20 16:26:25.661432 | controller | -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" 2026-03-20 16:26:25.661436 | controller | ]\n # If we are running the container locally, skip following test\n if 2026-03-20 16:26:25.661440 | controller | [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" 2026-03-20 16:26:25.661445 | controller | == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set 2026-03-20 16:26:25.661449 | controller | env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) 2026-03-20 16:26:25.661453 | controller | -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; 2026-03-20 16:26:25.661457 | controller | do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo 2026-03-20 16:26:25.661461 | controller | ${i}\n counter=$((counter + 1))\n fi\n done\n if 2026-03-20 16:26:25.661465 | controller | [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables 2026-03-20 16:26:25.661470 | controller | found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT 2026-03-20 16:26:25.661485 | controller | variables found.\"\n fi\n # check that no watcher endpoint uses https\n 2026-03-20 16:26:25.661492 | controller | \ oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-20 16:26:25.661498 | controller | infra-optim | [ $(grep -c https) == 0 ]\n ]\n logger.go:42: 16:21:10 | 2026-03-20 16:26:25.661504 | controller | watcher-tls/5-disable-tls | + oc exec -n watcher-kuttl-default openstackclient -- 2026-03-20 16:26:25.661510 | controller | openstack service list -f value -c Name -c Type\n logger.go:42: 16:21:10 | watcher-tls/5-disable-tls 2026-03-20 16:26:25.661514 | controller | | ++ grep -c '^watcher'\n logger.go:42: 16:21:12 | watcher-tls/5-disable-tls 2026-03-20 16:26:25.661518 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:21:12 | watcher-tls/5-disable-tls | ++ 2026-03-20 16:26:25.661522 | controller | awk '{print $1}'\n logger.go:42: 16:21:12 | watcher-tls/5-disable-tls | ++ grep 2026-03-20 16:26:25.661527 | controller | watcher\n logger.go:42: 16:21:12 | watcher-tls/5-disable-tls | ++ oc exec -n 2026-03-20 16:26:25.661531 | controller | watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-20 16:26:25.661535 | controller | -c Type -c ID\n logger.go:42: 16:21:14 | watcher-tls/5-disable-tls | + SERVICEID=0bc8267151fd4fefbc0a3327396365a4\n 2026-03-20 16:26:25.661539 | controller | \ logger.go:42: 16:21:14 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.661543 | controller | keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n logger.go:42: 16:21:15 2026-03-20 16:26:25.661547 | controller | | watcher-tls/5-disable-tls | + '[' 0bc8267151fd4fefbc0a3327396365a4 == 0bc8267151fd4fefbc0a3327396365a4 2026-03-20 16:26:25.661554 | controller | ']'\n logger.go:42: 16:21:15 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.661558 | controller | watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n logger.go:42: 16:21:15 2026-03-20 16:26:25.661562 | controller | | watcher-tls/5-disable-tls | + '[' -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.661567 | controller | ']'\n logger.go:42: 16:21:15 | watcher-tls/5-disable-tls | ++ oc get pods -n 2026-03-20 16:26:25.661571 | controller | openstack-operators -o name -l openstack.org/operator-name=watcher\n logger.go:42: 2026-03-20 16:26:25.661575 | controller | 16:21:15 | watcher-tls/5-disable-tls | + '[' '' == '' ']'\n logger.go:42: 16:21:15 2026-03-20 16:26:25.661579 | controller | | watcher-tls/5-disable-tls | + exit 0\n logger.go:42: 16:21:16 | watcher-tls/5-disable-tls 2026-03-20 16:26:25.661583 | controller | | running command: [sh -c set -euxo pipefail\n oc exec -n ${NAMESPACE} openstackclient 2026-03-20 16:26:25.661588 | controller | -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n 2026-03-20 16:26:25.661594 | controller | \ SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-20 16:26:25.661598 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ 2026-03-20 16:26:25.661602 | controller | $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.661606 | controller | == ${SERVICEID} ]\n [ -n \"$(oc get -n ${NAMESPACE} watcher watcher-kuttl 2026-03-20 16:26:25.661610 | controller | -o jsonpath={.status.hash.dbsync})\" ]\n # If we are running the container 2026-03-20 16:26:25.661615 | controller | locally, skip following test\n if [ \"$(oc get pods -n openstack-operators 2026-03-20 16:26:25.661619 | controller | -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 2026-03-20 16:26:25.661623 | controller | 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators 2026-03-20 16:26:25.661627 | controller | -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n 2026-03-20 16:26:25.661631 | controller | \ counter=0\n for i in ${env_variables}; do\n if echo ${i} 2026-03-20 16:26:25.661635 | controller | | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter 2026-03-20 16:26:25.661639 | controller | + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo 2026-03-20 16:26:25.661644 | controller | \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n 2026-03-20 16:26:25.661648 | controller | \ echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n 2026-03-20 16:26:25.661652 | controller | \ # check that no watcher endpoint uses https\n oc exec -n ${NAMESPACE} 2026-03-20 16:26:25.661656 | controller | openstackclient -- openstack endpoint list | grep infra-optim | [ $(grep -c https) 2026-03-20 16:26:25.661660 | controller | == 0 ]\n ]\n logger.go:42: 16:21:16 | watcher-tls/5-disable-tls | + oc 2026-03-20 16:26:25.661664 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-20 16:26:25.661669 | controller | -c Name -c Type\n logger.go:42: 16:21:16 | watcher-tls/5-disable-tls | ++ grep 2026-03-20 16:26:25.661673 | controller | -c '^watcher'\n logger.go:42: 16:21:18 | watcher-tls/5-disable-tls | + '[' 1 2026-03-20 16:26:25.661686 | controller | == 1 ']'\n logger.go:42: 16:21:18 | watcher-tls/5-disable-tls | ++ grep watcher\n 2026-03-20 16:26:25.661690 | controller | \ logger.go:42: 16:21:18 | watcher-tls/5-disable-tls | ++ oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.661708 | controller | openstackclient -- openstack service list -f value -c Name -c Type -c ID\n logger.go:42: 2026-03-20 16:26:25.661713 | controller | 16:21:18 | watcher-tls/5-disable-tls | ++ awk '{print $1}'\n logger.go:42: 16:21:20 2026-03-20 16:26:25.661727 | controller | | watcher-tls/5-disable-tls | + SERVICEID=0bc8267151fd4fefbc0a3327396365a4\n logger.go:42: 2026-03-20 16:26:25.661731 | controller | 16:21:20 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default keystoneservice 2026-03-20 16:26:25.661735 | controller | watcher -o 'jsonpath={.status.serviceID}'\n logger.go:42: 16:21:20 | watcher-tls/5-disable-tls 2026-03-20 16:26:25.661740 | controller | | + '[' 0bc8267151fd4fefbc0a3327396365a4 == 0bc8267151fd4fefbc0a3327396365a4 ']'\n 2026-03-20 16:26:25.661744 | controller | \ logger.go:42: 16:21:20 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.661748 | controller | watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n logger.go:42: 16:21:20 2026-03-20 16:26:25.661752 | controller | | watcher-tls/5-disable-tls | + '[' -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.661756 | controller | ']'\n logger.go:42: 16:21:20 | watcher-tls/5-disable-tls | ++ oc get pods -n 2026-03-20 16:26:25.661761 | controller | openstack-operators -o name -l openstack.org/operator-name=watcher\n logger.go:42: 2026-03-20 16:26:25.661765 | controller | 16:21:20 | watcher-tls/5-disable-tls | + '[' '' == '' ']'\n logger.go:42: 16:21:20 2026-03-20 16:26:25.661769 | controller | | watcher-tls/5-disable-tls | + exit 0\n logger.go:42: 16:21:20 | watcher-tls/5-disable-tls 2026-03-20 16:26:25.661773 | controller | | test step completed 5-disable-tls\n logger.go:42: 16:21:20 | watcher-tls/6-cleanup-watcher 2026-03-20 16:26:25.661777 | controller | | starting test step 6-cleanup-watcher\n logger.go:42: 16:21:20 | watcher-tls/6-cleanup-watcher 2026-03-20 16:26:25.661781 | controller | | running command: [sh -c set -ex\n oc exec -n watcher-kuttl-default openstackclient 2026-03-20 16:26:25.661786 | controller | -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 2026-03-20 16:26:25.661790 | controller | ]\n ]\n logger.go:42: 16:21:20 | watcher-tls/6-cleanup-watcher | + oc 2026-03-20 16:26:25.661794 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-20 16:26:25.661798 | controller | -c Name -c Type\n logger.go:42: 16:21:20 | watcher-tls/6-cleanup-watcher | ++ 2026-03-20 16:26:25.661802 | controller | grep -c '^watcher'\n logger.go:42: 16:21:23 | watcher-tls/6-cleanup-watcher | 2026-03-20 16:26:25.661806 | controller | + '[' 1 == 0 ']'\n logger.go:42: 16:21:24 | watcher-tls/6-cleanup-watcher | running 2026-03-20 16:26:25.661826 | controller | command: [sh -c set -ex\n oc exec -n watcher-kuttl-default openstackclient 2026-03-20 16:26:25.661831 | controller | -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 2026-03-20 16:26:25.661835 | controller | ]\n ]\n logger.go:42: 16:21:24 | watcher-tls/6-cleanup-watcher | + oc 2026-03-20 16:26:25.661839 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-20 16:26:25.661843 | controller | -c Name -c Type\n logger.go:42: 16:21:24 | watcher-tls/6-cleanup-watcher | ++ 2026-03-20 16:26:25.661848 | controller | grep -c '^watcher'\n logger.go:42: 16:21:26 | watcher-tls/6-cleanup-watcher | 2026-03-20 16:26:25.661852 | controller | + '[' 0 == 0 ']'\n logger.go:42: 16:21:27 | watcher-tls/6-cleanup-watcher | test 2026-03-20 16:26:25.661856 | controller | step completed 6-cleanup-watcher\n logger.go:42: 16:21:27 | watcher-tls/7-cleanup-certs 2026-03-20 16:26:25.661860 | controller | | starting test step 7-cleanup-certs\n logger.go:42: 16:21:27 | watcher-tls/7-cleanup-certs 2026-03-20 16:26:25.661864 | controller | | test step completed 7-cleanup-certs\n logger.go:42: 16:21:27 | watcher-tls 2026-03-20 16:26:25.661870 | controller | | skipping kubernetes event logging\n=== CONT kuttl/harness/watcher-rmquser\n logger.go:42: 2026-03-20 16:26:25.661875 | controller | 16:21:27 | watcher-rmquser | Skipping creation of user-supplied namespace: watcher-kuttl-default\n 2026-03-20 16:26:25.661879 | controller | \ logger.go:42: 16:21:27 | watcher-rmquser/0-cleanup-watcher | starting test step 2026-03-20 16:26:25.661885 | controller | 0-cleanup-watcher\n logger.go:42: 16:21:27 | watcher-rmquser/0-cleanup-watcher 2026-03-20 16:26:25.661902 | controller | | test step completed 0-cleanup-watcher\n logger.go:42: 16:21:27 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.661907 | controller | | starting test step 1-deploy\n logger.go:42: 16:21:27 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.661911 | controller | | Watcher:watcher-kuttl-default/watcher-kuttl created\n logger.go:42: 16:21:27 2026-03-20 16:26:25.661915 | controller | | watcher-rmquser/1-deploy | running command: [sh -c set -euxo pipefail\n \n 2026-03-20 16:26:25.661921 | controller | \ # Wait for Watcher to be Ready\n kubectl wait --for=condition=Ready 2026-03-20 16:26:25.661936 | controller | watcher/watcher-kuttl -n $NAMESPACE --timeout=300s\n \n # Verify WatcherNotificationTransportURLReady 2026-03-20 16:26:25.661943 | controller | condition exists and is True\n kubectl get watcher watcher-kuttl -n $NAMESPACE 2026-03-20 16:26:25.661949 | controller | -o jsonpath='{.status.conditions[?(@.type==\"WatcherNotificationTransportURLReady\")].status}' 2026-03-20 16:26:25.661955 | controller | | grep -q \"True\"\n echo \"WatcherNotificationTransportURLReady condition 2026-03-20 16:26:25.661969 | controller | is True\"\n \n # Count TransportURL CRs - should be exactly 2 (one 2026-03-20 16:26:25.661973 | controller | for messaging, one for notifications)\n transport_count=$(kubectl get transporturl 2026-03-20 16:26:25.661977 | controller | -n $NAMESPACE -o name | grep \"watcher-kuttl-watcher-transport\" | wc -l)\n notification_transport_count=$(kubectl 2026-03-20 16:26:25.661982 | controller | get transporturl -n $NAMESPACE -o name | grep \"watcher-kuttl-watcher-notification\" 2026-03-20 16:26:25.661986 | controller | | wc -l)\n \n if [ \"$transport_count\" -ne \"1\" ]; then\n echo 2026-03-20 16:26:25.661990 | controller | \"Expected 1 watcher-transport TransportURL, found $transport_count\"\n exit 2026-03-20 16:26:25.661994 | controller | 1\n fi\n \n if [ \"$notification_transport_count\" -ne \"1\" 2026-03-20 16:26:25.661998 | controller | ]; then\n echo \"Expected 1 notification-transport TransportURL, found 2026-03-20 16:26:25.662002 | controller | $notification_transport_count\"\n exit 1\n fi\n \n echo 2026-03-20 16:26:25.662007 | controller | \"Correctly found 2 TransportURLs (separate clusters: transport and notification)\"\n 2026-03-20 16:26:25.662011 | controller | \ \n # Verify watcher-transport has correct user and vhost\n transport_user=$(kubectl 2026-03-20 16:26:25.662015 | controller | get transporturl watcher-kuttl-watcher-transport -n $NAMESPACE -o jsonpath='{.spec.username}')\n 2026-03-20 16:26:25.662019 | controller | \ transport_vhost=$(kubectl get transporturl watcher-kuttl-watcher-transport 2026-03-20 16:26:25.662023 | controller | -n $NAMESPACE -o jsonpath='{.spec.vhost}')\n if [ \"$transport_user\" != 2026-03-20 16:26:25.662027 | controller | \"watcher-rpc\" ]; then\n echo \"Expected watcher-transport username 'watcher-rpc', 2026-03-20 16:26:25.662032 | controller | found '$transport_user'\"\n exit 1\n fi\n if [ \"$transport_vhost\" 2026-03-20 16:26:25.662036 | controller | != \"watcher-rpc\" ]; then\n echo \"Expected watcher-transport vhost 'watcher-rpc', 2026-03-20 16:26:25.662040 | controller | found '$transport_vhost'\"\n exit 1\n fi\n echo \"Watcher 2026-03-20 16:26:25.662052 | controller | transport has correct user (watcher-rpc) and vhost (watcher-rpc)\"\n \n # 2026-03-20 16:26:25.662056 | controller | Verify notification-transport has correct user and vhost\n notif_user=$(kubectl 2026-03-20 16:26:25.662060 | controller | get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications -n $NAMESPACE 2026-03-20 16:26:25.662064 | controller | -o jsonpath='{.spec.username}')\n notif_vhost=$(kubectl get transporturl 2026-03-20 16:26:25.662069 | controller | watcher-kuttl-watcher-notification-rabbitmq-notifications -n $NAMESPACE -o jsonpath='{.spec.vhost}')\n 2026-03-20 16:26:25.662078 | controller | \ if [ \"$notif_user\" != \"watcher-notifications\" ]; then\n echo 2026-03-20 16:26:25.662083 | controller | \"Expected notification-transport username 'watcher-notifications', found '$notif_user'\"\n 2026-03-20 16:26:25.662087 | controller | \ exit 1\n fi\n if [ \"$notif_vhost\" != \"watcher-notifications\" 2026-03-20 16:26:25.662091 | controller | ]; then\n echo \"Expected notification-transport vhost 'watcher-notifications', 2026-03-20 16:26:25.662095 | controller | found '$notif_vhost'\"\n exit 1\n fi\n echo \"Notification 2026-03-20 16:26:25.662099 | controller | transport has correct user (watcher-notifications) and vhost (watcher-notifications)\"\n 2026-03-20 16:26:25.662103 | controller | \ \n # Verify that watcher.conf contains the notifications transport_url\n 2026-03-20 16:26:25.662107 | controller | \ WATCHER_API_POD=$(kubectl get pods -n $NAMESPACE -l \"service=watcher-api\" 2026-03-20 16:26:25.662112 | controller | -o custom-columns=:metadata.name --no-headers | grep -v ^$ | head -1)\n if 2026-03-20 16:26:25.662116 | controller | [ -z \"${WATCHER_API_POD}\" ]; then\n echo \"No watcher-api pod found\"\n 2026-03-20 16:26:25.662120 | controller | \ exit 1\n fi\n # Verify RPC transport_url in DEFAULT section\n 2026-03-20 16:26:25.662124 | controller | \ rpc_transport_url=$(kubectl exec -n $NAMESPACE ${WATCHER_API_POD} -c watcher-api 2026-03-20 16:26:25.662137 | controller | -- cat /etc/watcher/watcher.conf.d/00-default.conf | grep -E '^\\[DEFAULT\\]' -A 2026-03-20 16:26:25.662141 | controller | 50 | grep 'transport_url' | head -1 || true)\n if [ -z \"$rpc_transport_url\" 2026-03-20 16:26:25.662146 | controller | ]; then\n echo \"transport_url not found in DEFAULT section\"\n exit 2026-03-20 16:26:25.662150 | controller | 1\n fi\n echo \"Found RPC transport_url: $rpc_transport_url\"\n \n 2026-03-20 16:26:25.662154 | controller | \ # Verify the RPC transport_url contains the correct vhost (watcher-rpc)\n 2026-03-20 16:26:25.662166 | controller | \ if ! echo \"$rpc_transport_url\" | grep -q '/watcher-rpc'; then\n echo 2026-03-20 16:26:25.662170 | controller | \"RPC transport_url does not contain expected vhost '/watcher-rpc'\"\n exit 2026-03-20 16:26:25.662175 | controller | 1\n fi\n echo \"Successfully verified vhost 'watcher-rpc' in RPC transport_url\"\n 2026-03-20 16:26:25.662180 | controller | \ \n # Verify the RPC transport_url contains the correct username (watcher-rpc)\n 2026-03-20 16:26:25.662184 | controller | \ if ! echo \"$rpc_transport_url\" | grep -q 'watcher-rpc:'; then\n echo 2026-03-20 16:26:25.662188 | controller | \"RPC transport_url does not contain expected username 'watcher-rpc:'\"\n exit 2026-03-20 16:26:25.662192 | controller | 1\n fi\n echo \"Successfully verified username 'watcher-rpc' in RPC 2026-03-20 16:26:25.662196 | controller | transport_url\"\n \n # Verify oslo_messaging_notifications section 2026-03-20 16:26:25.662201 | controller | has transport_url configured\n notif_transport_url=$(kubectl exec -n $NAMESPACE 2026-03-20 16:26:25.662205 | controller | ${WATCHER_API_POD} -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf 2026-03-20 16:26:25.662209 | controller | | grep -A 5 '\\[oslo_messaging_notifications\\]' | grep 'transport_url' || true)\n 2026-03-20 16:26:25.662213 | controller | \ if [ -z \"$notif_transport_url\" ]; then\n echo \"transport_url 2026-03-20 16:26:25.662217 | controller | not found in oslo_messaging_notifications section\"\n exit 1\n fi\n 2026-03-20 16:26:25.662221 | controller | \ echo \"Found notifications transport_url: $notif_transport_url\"\n \n 2026-03-20 16:26:25.662226 | controller | \ # Verify the notifications transport_url contains the correct vhost (watcher-notifications)\n 2026-03-20 16:26:25.662230 | controller | \ if ! echo \"$notif_transport_url\" | grep -q '/watcher-notifications'; then\n 2026-03-20 16:26:25.662234 | controller | \ echo \"Notifications transport_url does not contain expected vhost '/watcher-notifications'\"\n 2026-03-20 16:26:25.662240 | controller | \ exit 1\n fi\n echo \"Successfully verified vhost 'watcher-notifications' 2026-03-20 16:26:25.662244 | controller | in notifications transport_url\"\n \n # Verify the notifications transport_url 2026-03-20 16:26:25.662248 | controller | contains the correct username (watcher-notifications)\n if ! echo \"$notif_transport_url\" 2026-03-20 16:26:25.662253 | controller | | grep -q 'watcher-notifications:'; then\n echo \"Notifications transport_url 2026-03-20 16:26:25.662257 | controller | does not contain expected username 'watcher-notifications:'\"\n exit 1\n 2026-03-20 16:26:25.662264 | controller | \ fi\n echo \"Successfully verified username 'watcher-notifications' 2026-03-20 16:26:25.662268 | controller | in notifications transport_url\"\n \n exit 0\n ]\n logger.go:42: 2026-03-20 16:26:25.662272 | controller | 16:21:27 | watcher-rmquser/1-deploy | + kubectl wait --for=condition=Ready watcher/watcher-kuttl 2026-03-20 16:26:25.662277 | controller | -n watcher-kuttl-default --timeout=300s\n logger.go:42: 16:21:51 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.662281 | controller | | watcher.watcher.openstack.org/watcher-kuttl condition met\n logger.go:42: 16:21:51 2026-03-20 16:26:25.662285 | controller | | watcher-rmquser/1-deploy | + grep -q True\n logger.go:42: 16:21:51 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.662289 | controller | | + kubectl get watcher watcher-kuttl -n watcher-kuttl-default -o 'jsonpath={.status.conditions[?(@.type==\"WatcherNotificationTransportURLReady\")].status}'\n 2026-03-20 16:26:25.662293 | controller | \ logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | + echo 'WatcherNotificationTransportURLReady 2026-03-20 16:26:25.662297 | controller | condition is True'\n logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | WatcherNotificationTransportURLReady 2026-03-20 16:26:25.662302 | controller | condition is True\n logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | ++ grep 2026-03-20 16:26:25.662306 | controller | watcher-kuttl-watcher-transport\n logger.go:42: 16:21:52 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.662310 | controller | | ++ kubectl get transporturl -n watcher-kuttl-default -o name\n logger.go:42: 2026-03-20 16:26:25.662314 | controller | 16:21:52 | watcher-rmquser/1-deploy | ++ wc -l\n logger.go:42: 16:21:52 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.662318 | controller | | + transport_count=1\n logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | ++ 2026-03-20 16:26:25.662322 | controller | kubectl get transporturl -n watcher-kuttl-default -o name\n logger.go:42: 16:21:52 2026-03-20 16:26:25.662326 | controller | | watcher-rmquser/1-deploy | ++ wc -l\n logger.go:42: 16:21:52 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.662330 | controller | | ++ grep watcher-kuttl-watcher-notification\n logger.go:42: 16:21:52 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.662335 | controller | | + notification_transport_count=1\n logger.go:42: 16:21:52 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.662339 | controller | | + '[' 1 -ne 1 ']'\n logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | + '[' 2026-03-20 16:26:25.662343 | controller | 1 -ne 1 ']'\n logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | + echo 'Correctly 2026-03-20 16:26:25.662355 | controller | found 2 TransportURLs (separate clusters: transport and notification)'\n logger.go:42: 2026-03-20 16:26:25.662359 | controller | 16:21:52 | watcher-rmquser/1-deploy | Correctly found 2 TransportURLs (separate 2026-03-20 16:26:25.662363 | controller | clusters: transport and notification)\n logger.go:42: 16:21:52 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.662368 | controller | | ++ kubectl get transporturl watcher-kuttl-watcher-transport -n watcher-kuttl-default 2026-03-20 16:26:25.662372 | controller | -o 'jsonpath={.spec.username}'\n logger.go:42: 16:21:52 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.662376 | controller | | + transport_user=watcher-rpc\n logger.go:42: 16:21:52 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.662380 | controller | | ++ kubectl get transporturl watcher-kuttl-watcher-transport -n watcher-kuttl-default 2026-03-20 16:26:25.662389 | controller | -o 'jsonpath={.spec.vhost}'\n logger.go:42: 16:21:52 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.662393 | controller | | + transport_vhost=watcher-rpc\n logger.go:42: 16:21:52 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.662397 | controller | | + '[' watcher-rpc '!=' watcher-rpc ']'\n logger.go:42: 16:21:52 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.662403 | controller | | + '[' watcher-rpc '!=' watcher-rpc ']'\n logger.go:42: 16:21:52 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.662407 | controller | | + echo 'Watcher transport has correct user (watcher-rpc) and vhost (watcher-rpc)'\n 2026-03-20 16:26:25.662412 | controller | \ logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | Watcher transport has correct 2026-03-20 16:26:25.662424 | controller | user (watcher-rpc) and vhost (watcher-rpc)\n logger.go:42: 16:21:52 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.662428 | controller | | ++ kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications 2026-03-20 16:26:25.662432 | controller | -n watcher-kuttl-default -o 'jsonpath={.spec.username}'\n logger.go:42: 16:21:52 2026-03-20 16:26:25.662436 | controller | | watcher-rmquser/1-deploy | + notif_user=watcher-notifications\n logger.go:42: 2026-03-20 16:26:25.662441 | controller | 16:21:52 | watcher-rmquser/1-deploy | ++ kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications 2026-03-20 16:26:25.662445 | controller | -n watcher-kuttl-default -o 'jsonpath={.spec.vhost}'\n logger.go:42: 16:21:52 2026-03-20 16:26:25.662449 | controller | | watcher-rmquser/1-deploy | + notif_vhost=watcher-notifications\n logger.go:42: 2026-03-20 16:26:25.662453 | controller | 16:21:52 | watcher-rmquser/1-deploy | + '[' watcher-notifications '!=' watcher-notifications 2026-03-20 16:26:25.662457 | controller | ']'\n logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | + '[' watcher-notifications 2026-03-20 16:26:25.662462 | controller | '!=' watcher-notifications ']'\n logger.go:42: 16:21:52 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.662466 | controller | | + echo 'Notification transport has correct user (watcher-notifications) and vhost 2026-03-20 16:26:25.662470 | controller | (watcher-notifications)'\n logger.go:42: 16:21:52 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.662474 | controller | | Notification transport has correct user (watcher-notifications) and vhost (watcher-notifications)\n 2026-03-20 16:26:25.662478 | controller | \ logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | ++ grep -v '^$'\n logger.go:42: 2026-03-20 16:26:25.662483 | controller | 16:21:52 | watcher-rmquser/1-deploy | ++ head -1\n logger.go:42: 16:21:52 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.662487 | controller | | ++ kubectl get pods -n watcher-kuttl-default -l service=watcher-api -o custom-columns=:metadata.name 2026-03-20 16:26:25.662491 | controller | --no-headers\n logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | + WATCHER_API_POD=watcher-kuttl-api-0\n 2026-03-20 16:26:25.662495 | controller | \ logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | + '[' -z watcher-kuttl-api-0 2026-03-20 16:26:25.662499 | controller | ']'\n logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | ++ grep -E '^\\[DEFAULT\\]' 2026-03-20 16:26:25.662503 | controller | -A 50\n logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | ++ grep transport_url\n 2026-03-20 16:26:25.662508 | controller | \ logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | ++ kubectl exec -n watcher-kuttl-default 2026-03-20 16:26:25.662514 | controller | watcher-kuttl-api-0 -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-20 16:26:25.662518 | controller | \ logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | ++ head -1\n logger.go:42: 2026-03-20 16:26:25.662522 | controller | 16:21:53 | watcher-rmquser/1-deploy | + rpc_transport_url='transport_url = rabbit://watcher-rpc:3v320pCCv-k8XQA5ZC5ZgMG988bWL87e@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'\n 2026-03-20 16:26:25.662526 | controller | \ logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | + '[' -z 'transport_url 2026-03-20 16:26:25.662530 | controller | = rabbit://watcher-rpc:3v320pCCv-k8XQA5ZC5ZgMG988bWL87e@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1' 2026-03-20 16:26:25.662544 | controller | ']'\n logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | + echo 'Found RPC transport_url: 2026-03-20 16:26:25.662549 | controller | transport_url = rabbit://watcher-rpc:3v320pCCv-k8XQA5ZC5ZgMG988bWL87e@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'\n 2026-03-20 16:26:25.662561 | controller | \ logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | Found RPC transport_url: 2026-03-20 16:26:25.662565 | controller | transport_url = rabbit://watcher-rpc:3v320pCCv-k8XQA5ZC5ZgMG988bWL87e@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1\n 2026-03-20 16:26:25.662569 | controller | \ logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | + grep -q /watcher-rpc\n 2026-03-20 16:26:25.662574 | controller | \ logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | + echo 'transport_url = 2026-03-20 16:26:25.662578 | controller | rabbit://watcher-rpc:3v320pCCv-k8XQA5ZC5ZgMG988bWL87e@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'\n 2026-03-20 16:26:25.662582 | controller | \ logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | + echo 'Successfully verified 2026-03-20 16:26:25.662586 | controller | vhost '\\''watcher-rpc'\\'' in RPC transport_url'\n logger.go:42: 16:21:53 | 2026-03-20 16:26:25.662590 | controller | watcher-rmquser/1-deploy | Successfully verified vhost 'watcher-rpc' in RPC transport_url\n 2026-03-20 16:26:25.662595 | controller | \ logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | + grep -q watcher-rpc:\n 2026-03-20 16:26:25.662599 | controller | \ logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | + echo 'transport_url = 2026-03-20 16:26:25.662603 | controller | rabbit://watcher-rpc:3v320pCCv-k8XQA5ZC5ZgMG988bWL87e@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'\n 2026-03-20 16:26:25.662607 | controller | \ logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | + echo 'Successfully verified 2026-03-20 16:26:25.662611 | controller | username '\\''watcher-rpc'\\'' in RPC transport_url'\n logger.go:42: 16:21:53 2026-03-20 16:26:25.662616 | controller | | watcher-rmquser/1-deploy | Successfully verified username 'watcher-rpc' in RPC 2026-03-20 16:26:25.662620 | controller | transport_url\n logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | ++ grep -A 2026-03-20 16:26:25.662624 | controller | 5 '\\[oslo_messaging_notifications\\]'\n logger.go:42: 16:21:53 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.662628 | controller | | ++ kubectl exec -n watcher-kuttl-default watcher-kuttl-api-0 -c watcher-api -- 2026-03-20 16:26:25.662641 | controller | cat /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 16:21:53 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.662645 | controller | | ++ grep transport_url\n logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | 2026-03-20 16:26:25.662650 | controller | + notif_transport_url='transport_url = rabbit://watcher-notifications:S1IIyezEXhXsAVBQzExoKD685nXbbjCW@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'\n 2026-03-20 16:26:25.662654 | controller | \ logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | + '[' -z 'transport_url 2026-03-20 16:26:25.662658 | controller | = rabbit://watcher-notifications:S1IIyezEXhXsAVBQzExoKD685nXbbjCW@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1' 2026-03-20 16:26:25.662662 | controller | ']'\n logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | + echo 'Found notifications 2026-03-20 16:26:25.662667 | controller | transport_url: transport_url = rabbit://watcher-notifications:S1IIyezEXhXsAVBQzExoKD685nXbbjCW@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'\n 2026-03-20 16:26:25.662671 | controller | \ logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | Found notifications transport_url: 2026-03-20 16:26:25.662675 | controller | transport_url = rabbit://watcher-notifications:S1IIyezEXhXsAVBQzExoKD685nXbbjCW@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1\n 2026-03-20 16:26:25.662679 | controller | \ logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | + grep -q /watcher-notifications\n 2026-03-20 16:26:25.662686 | controller | \ logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | + echo 'transport_url = 2026-03-20 16:26:25.662691 | controller | rabbit://watcher-notifications:S1IIyezEXhXsAVBQzExoKD685nXbbjCW@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'\n 2026-03-20 16:26:25.662710 | controller | \ logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | + echo 'Successfully verified 2026-03-20 16:26:25.662714 | controller | vhost '\\''watcher-notifications'\\'' in notifications transport_url'\n logger.go:42: 2026-03-20 16:26:25.662719 | controller | 16:21:53 | watcher-rmquser/1-deploy | Successfully verified vhost 'watcher-notifications' 2026-03-20 16:26:25.662723 | controller | in notifications transport_url\n logger.go:42: 16:21:53 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.662727 | controller | | + grep -q watcher-notifications:\n logger.go:42: 16:21:53 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.662731 | controller | | + echo 'transport_url = rabbit://watcher-notifications:S1IIyezEXhXsAVBQzExoKD685nXbbjCW@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'\n 2026-03-20 16:26:25.662735 | controller | \ logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | + echo 'Successfully verified 2026-03-20 16:26:25.662740 | controller | username '\\''watcher-notifications'\\'' in notifications transport_url'\n logger.go:42: 2026-03-20 16:26:25.662744 | controller | 16:21:53 | watcher-rmquser/1-deploy | Successfully verified username 'watcher-notifications' 2026-03-20 16:26:25.662756 | controller | in notifications transport_url\n logger.go:42: 16:21:53 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.662760 | controller | | + exit 0\n logger.go:42: 16:21:54 | watcher-rmquser/1-deploy | running command: 2026-03-20 16:26:25.662764 | controller | [sh -c set -euxo pipefail\n \n # Wait for Watcher to be Ready\n kubectl 2026-03-20 16:26:25.662768 | controller | wait --for=condition=Ready watcher/watcher-kuttl -n $NAMESPACE --timeout=300s\n 2026-03-20 16:26:25.662773 | controller | \ \n # Verify WatcherNotificationTransportURLReady condition exists 2026-03-20 16:26:25.662777 | controller | and is True\n kubectl get watcher watcher-kuttl -n $NAMESPACE -o jsonpath='{.status.conditions[?(@.type==\"WatcherNotificationTransportURLReady\")].status}' 2026-03-20 16:26:25.662781 | controller | | grep -q \"True\"\n echo \"WatcherNotificationTransportURLReady condition 2026-03-20 16:26:25.662785 | controller | is True\"\n \n # Count TransportURL CRs - should be exactly 2 (one 2026-03-20 16:26:25.662789 | controller | for messaging, one for notifications)\n transport_count=$(kubectl get transporturl 2026-03-20 16:26:25.662793 | controller | -n $NAMESPACE -o name | grep \"watcher-kuttl-watcher-transport\" | wc -l)\n notification_transport_count=$(kubectl 2026-03-20 16:26:25.662798 | controller | get transporturl -n $NAMESPACE -o name | grep \"watcher-kuttl-watcher-notification\" 2026-03-20 16:26:25.662802 | controller | | wc -l)\n \n if [ \"$transport_count\" -ne \"1\" ]; then\n echo 2026-03-20 16:26:25.662809 | controller | \"Expected 1 watcher-transport TransportURL, found $transport_count\"\n exit 2026-03-20 16:26:25.662813 | controller | 1\n fi\n \n if [ \"$notification_transport_count\" -ne \"1\" 2026-03-20 16:26:25.662817 | controller | ]; then\n echo \"Expected 1 notification-transport TransportURL, found 2026-03-20 16:26:25.662821 | controller | $notification_transport_count\"\n exit 1\n fi\n \n echo 2026-03-20 16:26:25.662825 | controller | \"Correctly found 2 TransportURLs (separate clusters: transport and notification)\"\n 2026-03-20 16:26:25.662830 | controller | \ \n # Verify watcher-transport has correct user and vhost\n transport_user=$(kubectl 2026-03-20 16:26:25.662834 | controller | get transporturl watcher-kuttl-watcher-transport -n $NAMESPACE -o jsonpath='{.spec.username}')\n 2026-03-20 16:26:25.662838 | controller | \ transport_vhost=$(kubectl get transporturl watcher-kuttl-watcher-transport 2026-03-20 16:26:25.662842 | controller | -n $NAMESPACE -o jsonpath='{.spec.vhost}')\n if [ \"$transport_user\" != 2026-03-20 16:26:25.662855 | controller | \"watcher-rpc\" ]; then\n echo \"Expected watcher-transport username 'watcher-rpc', 2026-03-20 16:26:25.662859 | controller | found '$transport_user'\"\n exit 1\n fi\n if [ \"$transport_vhost\" 2026-03-20 16:26:25.662863 | controller | != \"watcher-rpc\" ]; then\n echo \"Expected watcher-transport vhost 'watcher-rpc', 2026-03-20 16:26:25.662868 | controller | found '$transport_vhost'\"\n exit 1\n fi\n echo \"Watcher 2026-03-20 16:26:25.662872 | controller | transport has correct user (watcher-rpc) and vhost (watcher-rpc)\"\n \n # 2026-03-20 16:26:25.662876 | controller | Verify notification-transport has correct user and vhost\n notif_user=$(kubectl 2026-03-20 16:26:25.662880 | controller | get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications -n $NAMESPACE 2026-03-20 16:26:25.662884 | controller | -o jsonpath='{.spec.username}')\n notif_vhost=$(kubectl get transporturl 2026-03-20 16:26:25.662888 | controller | watcher-kuttl-watcher-notification-rabbitmq-notifications -n $NAMESPACE -o jsonpath='{.spec.vhost}')\n 2026-03-20 16:26:25.662893 | controller | \ if [ \"$notif_user\" != \"watcher-notifications\" ]; then\n echo 2026-03-20 16:26:25.662897 | controller | \"Expected notification-transport username 'watcher-notifications', found '$notif_user'\"\n 2026-03-20 16:26:25.662901 | controller | \ exit 1\n fi\n if [ \"$notif_vhost\" != \"watcher-notifications\" 2026-03-20 16:26:25.662905 | controller | ]; then\n echo \"Expected notification-transport vhost 'watcher-notifications', 2026-03-20 16:26:25.662909 | controller | found '$notif_vhost'\"\n exit 1\n fi\n echo \"Notification 2026-03-20 16:26:25.662913 | controller | transport has correct user (watcher-notifications) and vhost (watcher-notifications)\"\n 2026-03-20 16:26:25.662918 | controller | \ \n # Verify that watcher.conf contains the notifications transport_url\n 2026-03-20 16:26:25.662922 | controller | \ WATCHER_API_POD=$(kubectl get pods -n $NAMESPACE -l \"service=watcher-api\" 2026-03-20 16:26:25.662926 | controller | -o custom-columns=:metadata.name --no-headers | grep -v ^$ | head -1)\n if 2026-03-20 16:26:25.662930 | controller | [ -z \"${WATCHER_API_POD}\" ]; then\n echo \"No watcher-api pod found\"\n 2026-03-20 16:26:25.662934 | controller | \ exit 1\n fi\n # Verify RPC transport_url in DEFAULT section\n 2026-03-20 16:26:25.662938 | controller | \ rpc_transport_url=$(kubectl exec -n $NAMESPACE ${WATCHER_API_POD} -c watcher-api 2026-03-20 16:26:25.662942 | controller | -- cat /etc/watcher/watcher.conf.d/00-default.conf | grep -E '^\\[DEFAULT\\]' -A 2026-03-20 16:26:25.662955 | controller | 50 | grep 'transport_url' | head -1 || true)\n if [ -z \"$rpc_transport_url\" 2026-03-20 16:26:25.662959 | controller | ]; then\n echo \"transport_url not found in DEFAULT section\"\n exit 2026-03-20 16:26:25.662964 | controller | 1\n fi\n echo \"Found RPC transport_url: $rpc_transport_url\"\n \n 2026-03-20 16:26:25.662968 | controller | \ # Verify the RPC transport_url contains the correct vhost (watcher-rpc)\n 2026-03-20 16:26:25.662972 | controller | \ if ! echo \"$rpc_transport_url\" | grep -q '/watcher-rpc'; then\n echo 2026-03-20 16:26:25.662976 | controller | \"RPC transport_url does not contain expected vhost '/watcher-rpc'\"\n exit 2026-03-20 16:26:25.662980 | controller | 1\n fi\n echo \"Successfully verified vhost 'watcher-rpc' in RPC transport_url\"\n 2026-03-20 16:26:25.662985 | controller | \ \n # Verify the RPC transport_url contains the correct username (watcher-rpc)\n 2026-03-20 16:26:25.662989 | controller | \ if ! echo \"$rpc_transport_url\" | grep -q 'watcher-rpc:'; then\n echo 2026-03-20 16:26:25.662995 | controller | \"RPC transport_url does not contain expected username 'watcher-rpc:'\"\n exit 2026-03-20 16:26:25.663000 | controller | 1\n fi\n echo \"Successfully verified username 'watcher-rpc' in RPC 2026-03-20 16:26:25.663006 | controller | transport_url\"\n \n # Verify oslo_messaging_notifications section 2026-03-20 16:26:25.663010 | controller | has transport_url configured\n notif_transport_url=$(kubectl exec -n $NAMESPACE 2026-03-20 16:26:25.663014 | controller | ${WATCHER_API_POD} -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf 2026-03-20 16:26:25.663018 | controller | | grep -A 5 '\\[oslo_messaging_notifications\\]' | grep 'transport_url' || true)\n 2026-03-20 16:26:25.663022 | controller | \ if [ -z \"$notif_transport_url\" ]; then\n echo \"transport_url 2026-03-20 16:26:25.663026 | controller | not found in oslo_messaging_notifications section\"\n exit 1\n fi\n 2026-03-20 16:26:25.663030 | controller | \ echo \"Found notifications transport_url: $notif_transport_url\"\n \n 2026-03-20 16:26:25.663036 | controller | \ # Verify the notifications transport_url contains the correct vhost (watcher-notifications)\n 2026-03-20 16:26:25.663041 | controller | \ if ! echo \"$notif_transport_url\" | grep -q '/watcher-notifications'; then\n 2026-03-20 16:26:25.663045 | controller | \ echo \"Notifications transport_url does not contain expected vhost '/watcher-notifications'\"\n 2026-03-20 16:26:25.663049 | controller | \ exit 1\n fi\n echo \"Successfully verified vhost 'watcher-notifications' 2026-03-20 16:26:25.663053 | controller | in notifications transport_url\"\n \n # Verify the notifications transport_url 2026-03-20 16:26:25.663057 | controller | contains the correct username (watcher-notifications)\n if ! echo \"$notif_transport_url\" 2026-03-20 16:26:25.663062 | controller | | grep -q 'watcher-notifications:'; then\n echo \"Notifications transport_url 2026-03-20 16:26:25.663066 | controller | does not contain expected username 'watcher-notifications:'\"\n exit 1\n 2026-03-20 16:26:25.663070 | controller | \ fi\n echo \"Successfully verified username 'watcher-notifications' 2026-03-20 16:26:25.663074 | controller | in notifications transport_url\"\n \n exit 0\n ]\n logger.go:42: 2026-03-20 16:26:25.663078 | controller | 16:21:54 | watcher-rmquser/1-deploy | + kubectl wait --for=condition=Ready watcher/watcher-kuttl 2026-03-20 16:26:25.663082 | controller | -n watcher-kuttl-default --timeout=300s\n logger.go:42: 16:21:54 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.663087 | controller | | watcher.watcher.openstack.org/watcher-kuttl condition met\n logger.go:42: 16:21:54 2026-03-20 16:26:25.663091 | controller | | watcher-rmquser/1-deploy | + kubectl get watcher watcher-kuttl -n watcher-kuttl-default 2026-03-20 16:26:25.663095 | controller | -o 'jsonpath={.status.conditions[?(@.type==\"WatcherNotificationTransportURLReady\")].status}'\n 2026-03-20 16:26:25.663100 | controller | \ logger.go:42: 16:21:54 | watcher-rmquser/1-deploy | + grep -q True\n logger.go:42: 2026-03-20 16:26:25.663104 | controller | 16:21:54 | watcher-rmquser/1-deploy | + echo 'WatcherNotificationTransportURLReady 2026-03-20 16:26:25.663108 | controller | condition is True'\n logger.go:42: 16:21:54 | watcher-rmquser/1-deploy | WatcherNotificationTransportURLReady 2026-03-20 16:26:25.663112 | controller | condition is True\n logger.go:42: 16:21:54 | watcher-rmquser/1-deploy | ++ grep 2026-03-20 16:26:25.663117 | controller | watcher-kuttl-watcher-transport\n logger.go:42: 16:21:54 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.663121 | controller | | ++ wc -l\n logger.go:42: 16:21:54 | watcher-rmquser/1-deploy | ++ kubectl get 2026-03-20 16:26:25.663125 | controller | transporturl -n watcher-kuttl-default -o name\n logger.go:42: 16:21:54 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.663129 | controller | | + transport_count=1\n logger.go:42: 16:21:54 | watcher-rmquser/1-deploy | ++ 2026-03-20 16:26:25.663133 | controller | grep watcher-kuttl-watcher-notification\n logger.go:42: 16:21:54 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.663138 | controller | | ++ wc -l\n logger.go:42: 16:21:54 | watcher-rmquser/1-deploy | ++ kubectl get 2026-03-20 16:26:25.663149 | controller | transporturl -n watcher-kuttl-default -o name\n logger.go:42: 16:21:55 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.663157 | controller | | + notification_transport_count=1\n logger.go:42: 16:21:55 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.663161 | controller | | + '[' 1 -ne 1 ']'\n logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | + '[' 2026-03-20 16:26:25.663165 | controller | 1 -ne 1 ']'\n logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | + echo 'Correctly 2026-03-20 16:26:25.663169 | controller | found 2 TransportURLs (separate clusters: transport and notification)'\n logger.go:42: 2026-03-20 16:26:25.663173 | controller | 16:21:55 | watcher-rmquser/1-deploy | Correctly found 2 TransportURLs (separate 2026-03-20 16:26:25.663177 | controller | clusters: transport and notification)\n logger.go:42: 16:21:55 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.663182 | controller | | ++ kubectl get transporturl watcher-kuttl-watcher-transport -n watcher-kuttl-default 2026-03-20 16:26:25.663186 | controller | -o 'jsonpath={.spec.username}'\n logger.go:42: 16:21:55 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.663190 | controller | | + transport_user=watcher-rpc\n logger.go:42: 16:21:55 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.663194 | controller | | ++ kubectl get transporturl watcher-kuttl-watcher-transport -n watcher-kuttl-default 2026-03-20 16:26:25.663198 | controller | -o 'jsonpath={.spec.vhost}'\n logger.go:42: 16:21:55 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.663202 | controller | | + transport_vhost=watcher-rpc\n logger.go:42: 16:21:55 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.663207 | controller | | + '[' watcher-rpc '!=' watcher-rpc ']'\n logger.go:42: 16:21:55 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.663211 | controller | | + '[' watcher-rpc '!=' watcher-rpc ']'\n logger.go:42: 16:21:55 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.663215 | controller | | + echo 'Watcher transport has correct user (watcher-rpc) and vhost (watcher-rpc)'\n 2026-03-20 16:26:25.663221 | controller | \ logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | Watcher transport has correct 2026-03-20 16:26:25.663225 | controller | user (watcher-rpc) and vhost (watcher-rpc)\n logger.go:42: 16:21:55 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.663229 | controller | | ++ kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications 2026-03-20 16:26:25.663234 | controller | -n watcher-kuttl-default -o 'jsonpath={.spec.username}'\n logger.go:42: 16:21:55 2026-03-20 16:26:25.663238 | controller | | watcher-rmquser/1-deploy | + notif_user=watcher-notifications\n logger.go:42: 2026-03-20 16:26:25.663242 | controller | 16:21:55 | watcher-rmquser/1-deploy | ++ kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications 2026-03-20 16:26:25.663246 | controller | -n watcher-kuttl-default -o 'jsonpath={.spec.vhost}'\n logger.go:42: 16:21:55 2026-03-20 16:26:25.663250 | controller | | watcher-rmquser/1-deploy | + notif_vhost=watcher-notifications\n logger.go:42: 2026-03-20 16:26:25.663254 | controller | 16:21:55 | watcher-rmquser/1-deploy | + '[' watcher-notifications '!=' watcher-notifications 2026-03-20 16:26:25.663259 | controller | ']'\n logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | + '[' watcher-notifications 2026-03-20 16:26:25.663263 | controller | '!=' watcher-notifications ']'\n logger.go:42: 16:21:55 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.663267 | controller | | + echo 'Notification transport has correct user (watcher-notifications) and vhost 2026-03-20 16:26:25.663271 | controller | (watcher-notifications)'\n logger.go:42: 16:21:55 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.663277 | controller | | Notification transport has correct user (watcher-notifications) and vhost (watcher-notifications)\n 2026-03-20 16:26:25.663282 | controller | \ logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | ++ grep -v '^$'\n logger.go:42: 2026-03-20 16:26:25.663288 | controller | 16:21:55 | watcher-rmquser/1-deploy | ++ kubectl get pods -n watcher-kuttl-default 2026-03-20 16:26:25.663294 | controller | -l service=watcher-api -o custom-columns=:metadata.name --no-headers\n logger.go:42: 2026-03-20 16:26:25.663299 | controller | 16:21:55 | watcher-rmquser/1-deploy | ++ head -1\n logger.go:42: 16:21:55 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.663306 | controller | | + WATCHER_API_POD=watcher-kuttl-api-0\n logger.go:42: 16:21:55 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.663310 | controller | | + '[' -z watcher-kuttl-api-0 ']'\n logger.go:42: 16:21:55 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.663314 | controller | | ++ head -1\n logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | ++ grep transport_url\n 2026-03-20 16:26:25.663319 | controller | \ logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | ++ grep -E '^\\[DEFAULT\\]' 2026-03-20 16:26:25.663323 | controller | -A 50\n logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | ++ kubectl exec -n 2026-03-20 16:26:25.663327 | controller | watcher-kuttl-default watcher-kuttl-api-0 -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-20 16:26:25.663331 | controller | \ logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | + rpc_transport_url='transport_url 2026-03-20 16:26:25.663335 | controller | = rabbit://watcher-rpc:3v320pCCv-k8XQA5ZC5ZgMG988bWL87e@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'\n 2026-03-20 16:26:25.663347 | controller | \ logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | + '[' -z 'transport_url 2026-03-20 16:26:25.663351 | controller | = rabbit://watcher-rpc:3v320pCCv-k8XQA5ZC5ZgMG988bWL87e@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1' 2026-03-20 16:26:25.663356 | controller | ']'\n logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | + echo 'Found RPC transport_url: 2026-03-20 16:26:25.663360 | controller | transport_url = rabbit://watcher-rpc:3v320pCCv-k8XQA5ZC5ZgMG988bWL87e@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'\n 2026-03-20 16:26:25.663364 | controller | \ logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | Found RPC transport_url: 2026-03-20 16:26:25.663368 | controller | transport_url = rabbit://watcher-rpc:3v320pCCv-k8XQA5ZC5ZgMG988bWL87e@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1\n 2026-03-20 16:26:25.663372 | controller | \ logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | + echo 'transport_url = 2026-03-20 16:26:25.663377 | controller | rabbit://watcher-rpc:3v320pCCv-k8XQA5ZC5ZgMG988bWL87e@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'\n 2026-03-20 16:26:25.663381 | controller | \ logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | + grep -q /watcher-rpc\n 2026-03-20 16:26:25.663385 | controller | \ logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | + echo 'Successfully verified 2026-03-20 16:26:25.663389 | controller | vhost '\\''watcher-rpc'\\'' in RPC transport_url'\n logger.go:42: 16:21:55 | 2026-03-20 16:26:25.663405 | controller | watcher-rmquser/1-deploy | Successfully verified vhost 'watcher-rpc' in RPC transport_url\n 2026-03-20 16:26:25.663411 | controller | \ logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | + grep -q watcher-rpc:\n 2026-03-20 16:26:25.663417 | controller | \ logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | + echo 'transport_url = 2026-03-20 16:26:25.663422 | controller | rabbit://watcher-rpc:3v320pCCv-k8XQA5ZC5ZgMG988bWL87e@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'\n 2026-03-20 16:26:25.663428 | controller | \ logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | + echo 'Successfully verified 2026-03-20 16:26:25.663434 | controller | username '\\''watcher-rpc'\\'' in RPC transport_url'\n logger.go:42: 16:21:55 2026-03-20 16:26:25.663438 | controller | | watcher-rmquser/1-deploy | Successfully verified username 'watcher-rpc' in RPC 2026-03-20 16:26:25.663443 | controller | transport_url\n logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | ++ grep -A 2026-03-20 16:26:25.663449 | controller | 5 '\\[oslo_messaging_notifications\\]'\n logger.go:42: 16:21:55 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.663454 | controller | | ++ grep transport_url\n logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | 2026-03-20 16:26:25.663458 | controller | ++ kubectl exec -n watcher-kuttl-default watcher-kuttl-api-0 -c watcher-api -- cat 2026-03-20 16:26:25.663471 | controller | /etc/watcher/watcher.conf.d/00-default.conf\n logger.go:42: 16:21:56 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.663475 | controller | | + notif_transport_url='transport_url = rabbit://watcher-notifications:S1IIyezEXhXsAVBQzExoKD685nXbbjCW@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'\n 2026-03-20 16:26:25.663482 | controller | \ logger.go:42: 16:21:56 | watcher-rmquser/1-deploy | + '[' -z 'transport_url 2026-03-20 16:26:25.663487 | controller | = rabbit://watcher-notifications:S1IIyezEXhXsAVBQzExoKD685nXbbjCW@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1' 2026-03-20 16:26:25.663493 | controller | ']'\n logger.go:42: 16:21:56 | watcher-rmquser/1-deploy | + echo 'Found notifications 2026-03-20 16:26:25.663498 | controller | transport_url: transport_url = rabbit://watcher-notifications:S1IIyezEXhXsAVBQzExoKD685nXbbjCW@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'\n 2026-03-20 16:26:25.663504 | controller | \ logger.go:42: 16:21:56 | watcher-rmquser/1-deploy | Found notifications transport_url: 2026-03-20 16:26:25.663510 | controller | transport_url = rabbit://watcher-notifications:S1IIyezEXhXsAVBQzExoKD685nXbbjCW@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1\n 2026-03-20 16:26:25.663516 | controller | \ logger.go:42: 16:21:56 | watcher-rmquser/1-deploy | + grep -q /watcher-notifications\n 2026-03-20 16:26:25.663521 | controller | \ logger.go:42: 16:21:56 | watcher-rmquser/1-deploy | + echo 'transport_url = 2026-03-20 16:26:25.663527 | controller | rabbit://watcher-notifications:S1IIyezEXhXsAVBQzExoKD685nXbbjCW@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'\n 2026-03-20 16:26:25.663547 | controller | \ logger.go:42: 16:21:56 | watcher-rmquser/1-deploy | + echo 'Successfully verified 2026-03-20 16:26:25.663553 | controller | vhost '\\''watcher-notifications'\\'' in notifications transport_url'\n logger.go:42: 2026-03-20 16:26:25.663574 | controller | 16:21:56 | watcher-rmquser/1-deploy | Successfully verified vhost 'watcher-notifications' 2026-03-20 16:26:25.663580 | controller | in notifications transport_url\n logger.go:42: 16:21:56 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.663584 | controller | | + grep -q watcher-notifications:\n logger.go:42: 16:21:56 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.663597 | controller | | + echo 'transport_url = rabbit://watcher-notifications:S1IIyezEXhXsAVBQzExoKD685nXbbjCW@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'\n 2026-03-20 16:26:25.663601 | controller | \ logger.go:42: 16:21:56 | watcher-rmquser/1-deploy | + echo 'Successfully verified 2026-03-20 16:26:25.663606 | controller | username '\\''watcher-notifications'\\'' in notifications transport_url'\n logger.go:42: 2026-03-20 16:26:25.663610 | controller | 16:21:56 | watcher-rmquser/1-deploy | Successfully verified username 'watcher-notifications' 2026-03-20 16:26:25.663614 | controller | in notifications transport_url\n logger.go:42: 16:21:56 | watcher-rmquser/1-deploy 2026-03-20 16:26:25.663618 | controller | | + exit 0\n logger.go:42: 16:21:56 | watcher-rmquser/1-deploy | test step completed 2026-03-20 16:26:25.663623 | controller | 1-deploy\n logger.go:42: 16:21:56 | watcher-rmquser/2-cleanup-watcher | starting 2026-03-20 16:26:25.663627 | controller | test step 2-cleanup-watcher\n logger.go:42: 16:22:01 | watcher-rmquser/2-cleanup-watcher 2026-03-20 16:26:25.663631 | controller | | test step completed 2-cleanup-watcher\n logger.go:42: 16:22:01 | watcher-rmquser 2026-03-20 16:26:25.663635 | controller | | skipping kubernetes event logging\n=== CONT kuttl/harness/watcher\n logger.go:42: 2026-03-20 16:26:25.663639 | controller | 16:22:01 | watcher | Skipping creation of user-supplied namespace: watcher-kuttl-default\n 2026-03-20 16:26:25.663643 | controller | \ logger.go:42: 16:22:01 | watcher/0-cleanup-watcher | starting test step 0-cleanup-watcher\n 2026-03-20 16:26:25.663648 | controller | \ logger.go:42: 16:22:01 | watcher/0-cleanup-watcher | test step completed 0-cleanup-watcher\n 2026-03-20 16:26:25.663652 | controller | \ logger.go:42: 16:22:01 | watcher/1-deploy-with-defaults | starting test step 2026-03-20 16:26:25.663656 | controller | 1-deploy-with-defaults\n logger.go:42: 16:22:02 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.663663 | controller | | Watcher:watcher-kuttl-default/watcher-kuttl created\n logger.go:42: 16:22:02 2026-03-20 16:26:25.663667 | controller | | watcher/1-deploy-with-defaults | running command: [sh -c set -euxo pipefail\n 2026-03-20 16:26:25.663671 | controller | \ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.663675 | controller | -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc exec 2026-03-20 16:26:25.663679 | controller | -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-20 16:26:25.663684 | controller | -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n watcher-kuttl-default 2026-03-20 16:26:25.663688 | controller | keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ 2026-03-20 16:26:25.663708 | controller | -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" 2026-03-20 16:26:25.663716 | controller | ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.663720 | controller | -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" == 1 ]\n [ \"$(oc 2026-03-20 16:26:25.663724 | controller | get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64 2026-03-20 16:26:25.663728 | controller | -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" == 2 2026-03-20 16:26:25.663732 | controller | ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.663737 | controller | -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'region_name = regionOne')\" 2026-03-20 16:26:25.663741 | controller | -ge 1 ]\n # If we are running the container locally, skip following test\n 2026-03-20 16:26:25.663745 | controller | \ if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" 2026-03-20 16:26:25.663750 | controller | == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set 2026-03-20 16:26:25.663754 | controller | env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) 2026-03-20 16:26:25.663759 | controller | -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; 2026-03-20 16:26:25.663763 | controller | do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo 2026-03-20 16:26:25.663767 | controller | ${i}\n counter=$((counter + 1))\n fi\n done\n if 2026-03-20 16:26:25.663771 | controller | [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables 2026-03-20 16:26:25.663775 | controller | found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT 2026-03-20 16:26:25.663779 | controller | variables found.\"\n fi\n ]\n logger.go:42: 16:22:02 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.663784 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-20 16:26:25.663788 | controller | value -c Name -c Type\n logger.go:42: 16:22:02 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.663792 | controller | | ++ grep -c '^watcher'\n logger.go:42: 16:22:04 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.663798 | controller | | + '[' 0 == 1 ']'\n logger.go:42: 16:22:05 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.663811 | controller | | running command: [sh -c set -euxo pipefail\n oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.663816 | controller | openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c 2026-03-20 16:26:25.663820 | controller | ^watcher) == 1 ]\n SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient 2026-03-20 16:26:25.663824 | controller | -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print 2026-03-20 16:26:25.663828 | controller | $1}')\n [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.663833 | controller | == ${SERVICEID} ]\n [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl 2026-03-20 16:26:25.663841 | controller | -o jsonpath={.status.hash.dbsync})\" ]\n [ \"$(oc get -n watcher-kuttl-default 2026-03-20 16:26:25.663845 | controller | secret watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep 2026-03-20 16:26:25.663849 | controller | -c 'ssl=1')\" == 1 ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.663854 | controller | -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" 2026-03-20 16:26:25.663858 | controller | == 2 ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.663862 | controller | -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'region_name = regionOne')\" 2026-03-20 16:26:25.663866 | controller | -ge 1 ]\n # If we are running the container locally, skip following test\n 2026-03-20 16:26:25.663870 | controller | \ if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" 2026-03-20 16:26:25.663875 | controller | == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set 2026-03-20 16:26:25.663879 | controller | env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) 2026-03-20 16:26:25.663883 | controller | -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; 2026-03-20 16:26:25.663887 | controller | do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo 2026-03-20 16:26:25.663891 | controller | ${i}\n counter=$((counter + 1))\n fi\n done\n if 2026-03-20 16:26:25.663895 | controller | [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables 2026-03-20 16:26:25.663900 | controller | found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT 2026-03-20 16:26:25.663904 | controller | variables found.\"\n fi\n ]\n logger.go:42: 16:22:05 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.663908 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-20 16:26:25.663912 | controller | value -c Name -c Type\n logger.go:42: 16:22:05 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.663916 | controller | | ++ grep -c '^watcher'\n logger.go:42: 16:22:07 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.663920 | controller | | + '[' 0 == 1 ']'\n logger.go:42: 16:22:08 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.663925 | controller | | running command: [sh -c set -euxo pipefail\n oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.663929 | controller | openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c 2026-03-20 16:26:25.663933 | controller | ^watcher) == 1 ]\n SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient 2026-03-20 16:26:25.663937 | controller | -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print 2026-03-20 16:26:25.663941 | controller | $1}')\n [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.663945 | controller | == ${SERVICEID} ]\n [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl 2026-03-20 16:26:25.663950 | controller | -o jsonpath={.status.hash.dbsync})\" ]\n [ \"$(oc get -n watcher-kuttl-default 2026-03-20 16:26:25.663954 | controller | secret watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep 2026-03-20 16:26:25.663958 | controller | -c 'ssl=1')\" == 1 ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.663962 | controller | -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" 2026-03-20 16:26:25.663967 | controller | == 2 ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.663971 | controller | -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'region_name = regionOne')\" 2026-03-20 16:26:25.663977 | controller | -ge 1 ]\n # If we are running the container locally, skip following test\n 2026-03-20 16:26:25.663983 | controller | \ if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" 2026-03-20 16:26:25.663989 | controller | == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set 2026-03-20 16:26:25.663993 | controller | env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) 2026-03-20 16:26:25.664005 | controller | -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; 2026-03-20 16:26:25.664010 | controller | do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo 2026-03-20 16:26:25.664014 | controller | ${i}\n counter=$((counter + 1))\n fi\n done\n if 2026-03-20 16:26:25.664018 | controller | [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables 2026-03-20 16:26:25.664023 | controller | found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT 2026-03-20 16:26:25.664027 | controller | variables found.\"\n fi\n ]\n logger.go:42: 16:22:08 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664031 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-20 16:26:25.664035 | controller | value -c Name -c Type\n logger.go:42: 16:22:08 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664039 | controller | | ++ grep -c '^watcher'\n logger.go:42: 16:22:10 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664043 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:22:10 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664049 | controller | | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.664054 | controller | -f value -c Name -c Type -c ID\n logger.go:42: 16:22:10 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664058 | controller | | ++ awk '{print $1}'\n logger.go:42: 16:22:10 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664062 | controller | | ++ grep watcher\n logger.go:42: 16:22:12 | watcher/1-deploy-with-defaults | 2026-03-20 16:26:25.664066 | controller | + SERVICEID=50cd2d8870524a79924faff92882dc56\n logger.go:42: 16:22:12 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664072 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-20 16:26:25.664076 | controller | \ logger.go:42: 16:22:13 | watcher/1-deploy-with-defaults | + '[' 50cd2d8870524a79924faff92882dc56 2026-03-20 16:26:25.664080 | controller | == 50cd2d8870524a79924faff92882dc56 ']'\n logger.go:42: 16:22:13 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664085 | controller | | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n 2026-03-20 16:26:25.664089 | controller | \ logger.go:42: 16:22:13 | watcher/1-deploy-with-defaults | + '[' -n '' ']'\n 2026-03-20 16:26:25.664093 | controller | \ logger.go:42: 16:22:14 | watcher/1-deploy-with-defaults | running command: [sh 2026-03-20 16:26:25.664097 | controller | -c set -euxo pipefail\n oc exec -n watcher-kuttl-default openstackclient 2026-03-20 16:26:25.664101 | controller | -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n 2026-03-20 16:26:25.664105 | controller | \ SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack 2026-03-20 16:26:25.664110 | controller | service list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n 2026-03-20 16:26:25.664114 | controller | \ [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.664118 | controller | == ${SERVICEID} ]\n [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl 2026-03-20 16:26:25.664132 | controller | -o jsonpath={.status.hash.dbsync})\" ]\n [ \"$(oc get -n watcher-kuttl-default 2026-03-20 16:26:25.664136 | controller | secret watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep 2026-03-20 16:26:25.664142 | controller | -c 'ssl=1')\" == 1 ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.664146 | controller | -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" 2026-03-20 16:26:25.664150 | controller | == 2 ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.664155 | controller | -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'region_name = regionOne')\" 2026-03-20 16:26:25.664159 | controller | -ge 1 ]\n # If we are running the container locally, skip following test\n 2026-03-20 16:26:25.664163 | controller | \ if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" 2026-03-20 16:26:25.664167 | controller | == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set 2026-03-20 16:26:25.664171 | controller | env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) 2026-03-20 16:26:25.664176 | controller | -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; 2026-03-20 16:26:25.664180 | controller | do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo 2026-03-20 16:26:25.664184 | controller | ${i}\n counter=$((counter + 1))\n fi\n done\n if 2026-03-20 16:26:25.664188 | controller | [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables 2026-03-20 16:26:25.664192 | controller | found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT 2026-03-20 16:26:25.664197 | controller | variables found.\"\n fi\n ]\n logger.go:42: 16:22:14 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664208 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-20 16:26:25.664212 | controller | value -c Name -c Type\n logger.go:42: 16:22:14 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664216 | controller | | ++ grep -c '^watcher'\n logger.go:42: 16:22:16 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664221 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:22:16 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664225 | controller | | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.664229 | controller | -f value -c Name -c Type -c ID\n logger.go:42: 16:22:16 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664233 | controller | | ++ grep watcher\n logger.go:42: 16:22:16 | watcher/1-deploy-with-defaults | 2026-03-20 16:26:25.664237 | controller | ++ awk '{print $1}'\n logger.go:42: 16:22:18 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664242 | controller | | + SERVICEID=50cd2d8870524a79924faff92882dc56\n logger.go:42: 16:22:18 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664246 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-20 16:26:25.664250 | controller | \ logger.go:42: 16:22:19 | watcher/1-deploy-with-defaults | + '[' 50cd2d8870524a79924faff92882dc56 2026-03-20 16:26:25.664254 | controller | == 50cd2d8870524a79924faff92882dc56 ']'\n logger.go:42: 16:22:19 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664258 | controller | | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n 2026-03-20 16:26:25.664263 | controller | \ logger.go:42: 16:22:19 | watcher/1-deploy-with-defaults | + '[' -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.664267 | controller | ']'\n logger.go:42: 16:22:19 | watcher/1-deploy-with-defaults | ++ base64 -d\n 2026-03-20 16:26:25.664271 | controller | \ logger.go:42: 16:22:19 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.664275 | controller | secret watcher-kuttl-api-config-data -o 'jsonpath={.data.my\\.cnf}'\n logger.go:42: 2026-03-20 16:26:25.664282 | controller | 16:22:19 | watcher/1-deploy-with-defaults | ++ grep -c ssl=1\n logger.go:42: 2026-03-20 16:26:25.664288 | controller | 16:22:19 | watcher/1-deploy-with-defaults | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-20 16:26:25.664292 | controller | 16:22:19 | watcher/1-deploy-with-defaults | ++ base64 -d\n logger.go:42: 16:22:19 2026-03-20 16:26:25.664297 | controller | | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.664301 | controller | -o 'jsonpath={.data.00-default\\.conf}'\n logger.go:42: 16:22:19 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664305 | controller | | ++ grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'\n logger.go:42: 2026-03-20 16:26:25.664309 | controller | 16:22:19 | watcher/1-deploy-with-defaults | + '[' 2 == 2 ']'\n logger.go:42: 2026-03-20 16:26:25.664313 | controller | 16:22:19 | watcher/1-deploy-with-defaults | ++ grep -c 'region_name = regionOne'\n 2026-03-20 16:26:25.664318 | controller | \ logger.go:42: 16:22:19 | watcher/1-deploy-with-defaults | ++ base64 -d\n logger.go:42: 2026-03-20 16:26:25.664322 | controller | 16:22:19 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret 2026-03-20 16:26:25.664326 | controller | watcher-kuttl-api-config-data -o 'jsonpath={.data.00-default\\.conf}'\n logger.go:42: 2026-03-20 16:26:25.664330 | controller | 16:22:19 | watcher/1-deploy-with-defaults | + '[' 9 -ge 1 ']'\n logger.go:42: 2026-03-20 16:26:25.664334 | controller | 16:22:19 | watcher/1-deploy-with-defaults | ++ oc get pods -n openstack-operators 2026-03-20 16:26:25.664339 | controller | -o name -l openstack.org/operator-name=watcher\n logger.go:42: 16:22:19 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664343 | controller | | + '[' '' == '' ']'\n logger.go:42: 16:22:19 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664347 | controller | | + exit 0\n logger.go:42: 16:22:20 | watcher/1-deploy-with-defaults | running 2026-03-20 16:26:25.664351 | controller | command: [sh -c set -euxo pipefail\n oc exec -n watcher-kuttl-default openstackclient 2026-03-20 16:26:25.664355 | controller | -- openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n 2026-03-20 16:26:25.664360 | controller | \ SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack 2026-03-20 16:26:25.664364 | controller | service list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n 2026-03-20 16:26:25.664368 | controller | \ [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.664374 | controller | == ${SERVICEID} ]\n [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl 2026-03-20 16:26:25.664378 | controller | -o jsonpath={.status.hash.dbsync})\" ]\n [ \"$(oc get -n watcher-kuttl-default 2026-03-20 16:26:25.664382 | controller | secret watcher-kuttl-api-config-data -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep 2026-03-20 16:26:25.664387 | controller | -c 'ssl=1')\" == 1 ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.664398 | controller | -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" 2026-03-20 16:26:25.664403 | controller | == 2 ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.664407 | controller | -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'region_name = regionOne')\" 2026-03-20 16:26:25.664411 | controller | -ge 1 ]\n # If we are running the container locally, skip following test\n 2026-03-20 16:26:25.664415 | controller | \ if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" 2026-03-20 16:26:25.664420 | controller | == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set 2026-03-20 16:26:25.664424 | controller | env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) 2026-03-20 16:26:25.664428 | controller | -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; 2026-03-20 16:26:25.664432 | controller | do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo 2026-03-20 16:26:25.664438 | controller | ${i}\n counter=$((counter + 1))\n fi\n done\n if 2026-03-20 16:26:25.664442 | controller | [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables 2026-03-20 16:26:25.664447 | controller | found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT 2026-03-20 16:26:25.664451 | controller | variables found.\"\n fi\n ]\n logger.go:42: 16:22:20 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664455 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-20 16:26:25.664459 | controller | value -c Name -c Type\n logger.go:42: 16:22:20 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664463 | controller | | ++ grep -c '^watcher'\n logger.go:42: 16:22:22 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664468 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:22:22 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664472 | controller | | ++ grep watcher\n logger.go:42: 16:22:22 | watcher/1-deploy-with-defaults | 2026-03-20 16:26:25.664476 | controller | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-20 16:26:25.664480 | controller | value -c Name -c Type -c ID\n logger.go:42: 16:22:22 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664484 | controller | | ++ awk '{print $1}'\n logger.go:42: 16:22:25 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664488 | controller | | + SERVICEID=50cd2d8870524a79924faff92882dc56\n logger.go:42: 16:22:25 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664493 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-20 16:26:25.664497 | controller | \ logger.go:42: 16:22:25 | watcher/1-deploy-with-defaults | + '[' 50cd2d8870524a79924faff92882dc56 2026-03-20 16:26:25.664501 | controller | == 50cd2d8870524a79924faff92882dc56 ']'\n logger.go:42: 16:22:25 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664505 | controller | | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n 2026-03-20 16:26:25.664509 | controller | \ logger.go:42: 16:22:25 | watcher/1-deploy-with-defaults | + '[' -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.664517 | controller | ']'\n logger.go:42: 16:22:25 | watcher/1-deploy-with-defaults | ++ grep -c ssl=1\n 2026-03-20 16:26:25.664521 | controller | \ logger.go:42: 16:22:25 | watcher/1-deploy-with-defaults | ++ base64 -d\n logger.go:42: 2026-03-20 16:26:25.664525 | controller | 16:22:25 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret 2026-03-20 16:26:25.664529 | controller | watcher-kuttl-api-config-data -o 'jsonpath={.data.my\\.cnf}'\n logger.go:42: 2026-03-20 16:26:25.664534 | controller | 16:22:25 | watcher/1-deploy-with-defaults | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-20 16:26:25.664538 | controller | 16:22:25 | watcher/1-deploy-with-defaults | ++ base64 -d\n logger.go:42: 16:22:25 2026-03-20 16:26:25.664542 | controller | | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.664546 | controller | -o 'jsonpath={.data.00-default\\.conf}'\n logger.go:42: 16:22:25 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664550 | controller | | ++ grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'\n logger.go:42: 2026-03-20 16:26:25.664555 | controller | 16:22:25 | watcher/1-deploy-with-defaults | + '[' 2 == 2 ']'\n logger.go:42: 2026-03-20 16:26:25.664559 | controller | 16:22:25 | watcher/1-deploy-with-defaults | ++ base64 -d\n logger.go:42: 16:22:25 2026-03-20 16:26:25.664563 | controller | | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.664567 | controller | -o 'jsonpath={.data.00-default\\.conf}'\n logger.go:42: 16:22:25 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664571 | controller | | ++ grep -c 'region_name = regionOne'\n logger.go:42: 16:22:25 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664584 | controller | | + '[' 9 -ge 1 ']'\n logger.go:42: 16:22:25 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664589 | controller | | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n 2026-03-20 16:26:25.664594 | controller | \ logger.go:42: 16:22:26 | watcher/1-deploy-with-defaults | + '[' '' == '' ']'\n 2026-03-20 16:26:25.664598 | controller | \ logger.go:42: 16:22:26 | watcher/1-deploy-with-defaults | + exit 0\n logger.go:42: 2026-03-20 16:26:25.664602 | controller | 16:22:27 | watcher/1-deploy-with-defaults | running command: [sh -c set -euxo pipefail\n 2026-03-20 16:26:25.664606 | controller | \ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.664610 | controller | -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc exec 2026-03-20 16:26:25.664615 | controller | -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-20 16:26:25.664619 | controller | -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n watcher-kuttl-default 2026-03-20 16:26:25.664623 | controller | keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ 2026-03-20 16:26:25.664627 | controller | -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" 2026-03-20 16:26:25.664632 | controller | ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.664636 | controller | -o jsonpath='{.data.my\\.cnf}'|base64 -d|grep -c 'ssl=1')\" == 1 ]\n [ \"$(oc 2026-03-20 16:26:25.664640 | controller | get -n watcher-kuttl-default secret watcher-kuttl-api-config-data -o jsonpath='{.data.00-default\\.conf}'|base64 2026-03-20 16:26:25.664644 | controller | -d|grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem')\" == 2 2026-03-20 16:26:25.664648 | controller | ]\n [ \"$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.664652 | controller | -o jsonpath='{.data.00-default\\.conf}'|base64 -d|grep -c 'region_name = regionOne')\" 2026-03-20 16:26:25.664657 | controller | -ge 1 ]\n # If we are running the container locally, skip following test\n 2026-03-20 16:26:25.664661 | controller | \ if [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" 2026-03-20 16:26:25.664665 | controller | == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set 2026-03-20 16:26:25.664669 | controller | env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) 2026-03-20 16:26:25.664673 | controller | -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; 2026-03-20 16:26:25.664678 | controller | do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo 2026-03-20 16:26:25.664682 | controller | ${i}\n counter=$((counter + 1))\n fi\n done\n if 2026-03-20 16:26:25.664686 | controller | [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables 2026-03-20 16:26:25.664690 | controller | found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT 2026-03-20 16:26:25.664708 | controller | variables found.\"\n fi\n ]\n logger.go:42: 16:22:27 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664712 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-20 16:26:25.664717 | controller | value -c Name -c Type\n logger.go:42: 16:22:27 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664721 | controller | | ++ grep -c '^watcher'\n logger.go:42: 16:22:29 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664725 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:22:29 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664729 | controller | | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.664733 | controller | -f value -c Name -c Type -c ID\n logger.go:42: 16:22:29 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664741 | controller | | ++ grep watcher\n logger.go:42: 16:22:29 | watcher/1-deploy-with-defaults | 2026-03-20 16:26:25.664746 | controller | ++ awk '{print $1}'\n logger.go:42: 16:22:31 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664750 | controller | | + SERVICEID=50cd2d8870524a79924faff92882dc56\n logger.go:42: 16:22:31 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664754 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-20 16:26:25.664758 | controller | \ logger.go:42: 16:22:31 | watcher/1-deploy-with-defaults | + '[' 50cd2d8870524a79924faff92882dc56 2026-03-20 16:26:25.664762 | controller | == 50cd2d8870524a79924faff92882dc56 ']'\n logger.go:42: 16:22:31 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664766 | controller | | ++ oc get -n watcher-kuttl-default watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n 2026-03-20 16:26:25.664770 | controller | \ logger.go:42: 16:22:31 | watcher/1-deploy-with-defaults | + '[' -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.664775 | controller | ']'\n logger.go:42: 16:22:31 | watcher/1-deploy-with-defaults | ++ oc get -n 2026-03-20 16:26:25.664787 | controller | watcher-kuttl-default secret watcher-kuttl-api-config-data -o 'jsonpath={.data.my\\.cnf}'\n 2026-03-20 16:26:25.664792 | controller | \ logger.go:42: 16:22:31 | watcher/1-deploy-with-defaults | ++ grep -c ssl=1\n 2026-03-20 16:26:25.664796 | controller | \ logger.go:42: 16:22:31 | watcher/1-deploy-with-defaults | ++ base64 -d\n logger.go:42: 2026-03-20 16:26:25.664800 | controller | 16:22:31 | watcher/1-deploy-with-defaults | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-20 16:26:25.664805 | controller | 16:22:31 | watcher/1-deploy-with-defaults | ++ base64 -d\n logger.go:42: 16:22:31 2026-03-20 16:26:25.664809 | controller | | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.664813 | controller | -o 'jsonpath={.data.00-default\\.conf}'\n logger.go:42: 16:22:31 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664817 | controller | | ++ grep -c 'cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'\n logger.go:42: 2026-03-20 16:26:25.664821 | controller | 16:22:31 | watcher/1-deploy-with-defaults | + '[' 2 == 2 ']'\n logger.go:42: 2026-03-20 16:26:25.664825 | controller | 16:22:31 | watcher/1-deploy-with-defaults | ++ base64 -d\n logger.go:42: 16:22:31 2026-03-20 16:26:25.664830 | controller | | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.664834 | controller | -o 'jsonpath={.data.00-default\\.conf}'\n logger.go:42: 16:22:31 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664838 | controller | | ++ grep -c 'region_name = regionOne'\n logger.go:42: 16:22:32 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664842 | controller | | + '[' 9 -ge 1 ']'\n logger.go:42: 16:22:32 | watcher/1-deploy-with-defaults 2026-03-20 16:26:25.664846 | controller | | ++ oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n 2026-03-20 16:26:25.664850 | controller | \ logger.go:42: 16:22:32 | watcher/1-deploy-with-defaults | + '[' '' == '' ']'\n 2026-03-20 16:26:25.664855 | controller | \ logger.go:42: 16:22:32 | watcher/1-deploy-with-defaults | + exit 0\n logger.go:42: 2026-03-20 16:26:25.664859 | controller | 16:22:32 | watcher/1-deploy-with-defaults | test step completed 1-deploy-with-defaults\n 2026-03-20 16:26:25.664863 | controller | \ logger.go:42: 16:22:32 | watcher/2-cleanup-watcher | starting test step 2-cleanup-watcher\n 2026-03-20 16:26:25.664867 | controller | \ logger.go:42: 16:22:32 | watcher/2-cleanup-watcher | test step completed 2-cleanup-watcher\n 2026-03-20 16:26:25.664871 | controller | \ logger.go:42: 16:22:32 | watcher/3-precreate-mariadbaccount | starting test 2026-03-20 16:26:25.664876 | controller | step 3-precreate-mariadbaccount\n logger.go:42: 16:22:32 | watcher/3-precreate-mariadbaccount 2026-03-20 16:26:25.664883 | controller | | MariaDBAccount:watcher-kuttl-default/watcher-precreated created\n logger.go:42: 2026-03-20 16:26:25.664887 | controller | 16:22:32 | watcher/3-precreate-mariadbaccount | test step completed 3-precreate-mariadbaccount\n 2026-03-20 16:26:25.664891 | controller | \ logger.go:42: 16:22:32 | watcher/4-deploy-with-precreated-account | starting 2026-03-20 16:26:25.664895 | controller | test step 4-deploy-with-precreated-account\n logger.go:42: 16:22:32 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.664900 | controller | | Secret:watcher-kuttl-default/custom-prometheus-config created\n logger.go:42: 2026-03-20 16:26:25.664904 | controller | 16:22:32 | watcher/4-deploy-with-precreated-account | Watcher:watcher-kuttl-default/watcher-kuttl 2026-03-20 16:26:25.664908 | controller | created\n logger.go:42: 16:22:32 | watcher/4-deploy-with-precreated-account | 2026-03-20 16:26:25.664912 | controller | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-20 16:26:25.664916 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-20 16:26:25.664920 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-20 16:26:25.664925 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.664929 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-20 16:26:25.664933 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-20 16:26:25.664937 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-20 16:26:25.664942 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-20 16:26:25.664946 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.664950 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-20 16:26:25.664954 | controller | \ logger.go:42: 16:22:32 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.664959 | controller | watcher-kuttl-default\n logger.go:42: 16:22:32 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.664969 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.664973 | controller | \ logger.go:42: 16:22:32 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.664978 | controller | -v '^$'\n logger.go:42: 16:22:32 | watcher/4-deploy-with-precreated-account | 2026-03-20 16:26:25.664991 | controller | ++ head -1\n logger.go:42: 16:22:32 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.664996 | controller | | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.665000 | controller | \ logger.go:42: 16:22:32 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0\n 2026-03-20 16:26:25.665004 | controller | \ logger.go:42: 16:22:32 | watcher/4-deploy-with-precreated-account | + '[' -n 2026-03-20 16:26:25.665008 | controller | watcher-kuttl-api-0 ']'\n logger.go:42: 16:22:32 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.665012 | controller | | ++ grep -c '^# Global config'\n logger.go:42: 16:22:32 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.665016 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n 2026-03-20 16:26:25.665021 | controller | \ logger.go:42: 16:22:34 | watcher/4-deploy-with-precreated-account | error: Internal 2026-03-20 16:26:25.665025 | controller | error occurred: error executing command in container: container is not created or 2026-03-20 16:26:25.665029 | controller | running\n logger.go:42: 16:22:34 | watcher/4-deploy-with-precreated-account | 2026-03-20 16:26:25.665037 | controller | ++ echo\n logger.go:42: 16:22:34 | watcher/4-deploy-with-precreated-account | 2026-03-20 16:26:25.665041 | controller | + '[' 0 == 1 ']'\n logger.go:42: 16:22:35 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.665045 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-20 16:26:25.665050 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-20 16:26:25.665063 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-20 16:26:25.665067 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.665072 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-20 16:26:25.665076 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-20 16:26:25.665080 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-20 16:26:25.665084 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-20 16:26:25.665089 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.665093 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-20 16:26:25.665097 | controller | \ logger.go:42: 16:22:35 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.665101 | controller | watcher-kuttl-default\n logger.go:42: 16:22:35 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.665105 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.665109 | controller | \ logger.go:42: 16:22:35 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-20 16:26:25.665114 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.665118 | controller | \ logger.go:42: 16:22:35 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.665122 | controller | -v '^$'\n logger.go:42: 16:22:35 | watcher/4-deploy-with-precreated-account | 2026-03-20 16:26:25.665126 | controller | ++ head -1\n logger.go:42: 16:22:35 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.665130 | controller | | + APIPOD=\n logger.go:42: 16:22:36 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.665134 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-20 16:26:25.665139 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-20 16:26:25.665143 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-20 16:26:25.665147 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.665151 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-20 16:26:25.665166 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-20 16:26:25.665171 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-20 16:26:25.665185 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-20 16:26:25.665192 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.665199 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-20 16:26:25.665203 | controller | \ logger.go:42: 16:22:36 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.665207 | controller | watcher-kuttl-default\n logger.go:42: 16:22:36 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.665211 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.665216 | controller | \ logger.go:42: 16:22:36 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-20 16:26:25.665220 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.665225 | controller | \ logger.go:42: 16:22:36 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-20 16:26:25.665230 | controller | -1\n logger.go:42: 16:22:36 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.665235 | controller | -v '^$'\n logger.go:42: 16:22:36 | watcher/4-deploy-with-precreated-account | 2026-03-20 16:26:25.665241 | controller | + APIPOD=\n logger.go:42: 16:22:37 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.665247 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-20 16:26:25.665252 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-20 16:26:25.665257 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-20 16:26:25.665261 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.665268 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-20 16:26:25.665272 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-20 16:26:25.665276 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-20 16:26:25.665283 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-20 16:26:25.665287 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.665291 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-20 16:26:25.665295 | controller | \ logger.go:42: 16:22:37 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.665300 | controller | watcher-kuttl-default\n logger.go:42: 16:22:38 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.665305 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.665309 | controller | \ logger.go:42: 16:22:38 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-20 16:26:25.665314 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.665318 | controller | \ logger.go:42: 16:22:38 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-20 16:26:25.665322 | controller | -1\n logger.go:42: 16:22:38 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.665326 | controller | -v '^$'\n logger.go:42: 16:22:38 | watcher/4-deploy-with-precreated-account | 2026-03-20 16:26:25.665330 | controller | + APIPOD=\n logger.go:42: 16:22:39 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.665335 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-20 16:26:25.665339 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-20 16:26:25.665343 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-20 16:26:25.665349 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.665353 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-20 16:26:25.665357 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-20 16:26:25.665361 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-20 16:26:25.665366 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-20 16:26:25.665378 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.665383 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-20 16:26:25.665387 | controller | \ logger.go:42: 16:22:39 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.665392 | controller | watcher-kuttl-default\n logger.go:42: 16:22:39 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.665396 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.665400 | controller | \ logger.go:42: 16:22:39 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-20 16:26:25.665404 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.665408 | controller | \ logger.go:42: 16:22:39 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-20 16:26:25.665412 | controller | -1\n logger.go:42: 16:22:39 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.665416 | controller | -v '^$'\n logger.go:42: 16:22:39 | watcher/4-deploy-with-precreated-account | 2026-03-20 16:26:25.665421 | controller | + APIPOD=\n logger.go:42: 16:22:40 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.665425 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-20 16:26:25.665429 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-20 16:26:25.665433 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-20 16:26:25.665437 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.665442 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-20 16:26:25.665446 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-20 16:26:25.665450 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-20 16:26:25.665454 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-20 16:26:25.665459 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.665463 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-20 16:26:25.665469 | controller | \ logger.go:42: 16:22:40 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.665473 | controller | watcher-kuttl-default\n logger.go:42: 16:22:40 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.665477 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.665481 | controller | \ logger.go:42: 16:22:40 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-20 16:26:25.665486 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.665494 | controller | \ logger.go:42: 16:22:40 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-20 16:26:25.665499 | controller | -1\n logger.go:42: 16:22:40 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.665503 | controller | -v '^$'\n logger.go:42: 16:22:41 | watcher/4-deploy-with-precreated-account | 2026-03-20 16:26:25.665507 | controller | + APIPOD=\n logger.go:42: 16:22:42 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.665511 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-20 16:26:25.665515 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-20 16:26:25.665520 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-20 16:26:25.665524 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.665528 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-20 16:26:25.665532 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-20 16:26:25.665537 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-20 16:26:25.665541 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-20 16:26:25.665545 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.665564 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-20 16:26:25.665569 | controller | \ logger.go:42: 16:22:42 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.665573 | controller | watcher-kuttl-default\n logger.go:42: 16:22:42 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.665579 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.665583 | controller | \ logger.go:42: 16:22:42 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-20 16:26:25.665587 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.665591 | controller | \ logger.go:42: 16:22:42 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-20 16:26:25.665595 | controller | -1\n logger.go:42: 16:22:42 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.665599 | controller | -v '^$'\n logger.go:42: 16:22:42 | watcher/4-deploy-with-precreated-account | 2026-03-20 16:26:25.665604 | controller | + APIPOD=\n logger.go:42: 16:22:43 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.665608 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-20 16:26:25.665612 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-20 16:26:25.665616 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-20 16:26:25.665620 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.665625 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-20 16:26:25.665629 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-20 16:26:25.665633 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-20 16:26:25.665637 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-20 16:26:25.665643 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.665648 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-20 16:26:25.665652 | controller | \ logger.go:42: 16:22:43 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.665656 | controller | watcher-kuttl-default\n logger.go:42: 16:22:43 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.665660 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.665664 | controller | \ logger.go:42: 16:22:43 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-20 16:26:25.665668 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.665673 | controller | \ logger.go:42: 16:22:43 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-20 16:26:25.665677 | controller | -1\n logger.go:42: 16:22:43 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.665681 | controller | -v '^$'\n logger.go:42: 16:22:43 | watcher/4-deploy-with-precreated-account | 2026-03-20 16:26:25.665685 | controller | + APIPOD=\n logger.go:42: 16:22:44 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.665706 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-20 16:26:25.665714 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-20 16:26:25.665718 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-20 16:26:25.665722 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.665726 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-20 16:26:25.665731 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-20 16:26:25.665735 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-20 16:26:25.665739 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-20 16:26:25.665743 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.665747 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-20 16:26:25.665752 | controller | \ logger.go:42: 16:22:44 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.665762 | controller | watcher-kuttl-default\n logger.go:42: 16:22:45 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.665767 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.665771 | controller | \ logger.go:42: 16:22:45 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-20 16:26:25.665776 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.665782 | controller | \ logger.go:42: 16:22:45 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-20 16:26:25.665787 | controller | -1\n logger.go:42: 16:22:45 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.665793 | controller | -v '^$'\n logger.go:42: 16:22:45 | watcher/4-deploy-with-precreated-account | 2026-03-20 16:26:25.665799 | controller | + APIPOD=\n logger.go:42: 16:22:46 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.665805 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-20 16:26:25.665813 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-20 16:26:25.665817 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-20 16:26:25.665821 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.665826 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-20 16:26:25.665830 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-20 16:26:25.665834 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-20 16:26:25.665838 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-20 16:26:25.665842 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.665847 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-20 16:26:25.665851 | controller | \ logger.go:42: 16:22:46 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.665855 | controller | watcher-kuttl-default\n logger.go:42: 16:22:46 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.665859 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.665864 | controller | \ logger.go:42: 16:22:46 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-20 16:26:25.665868 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.665872 | controller | \ logger.go:42: 16:22:46 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-20 16:26:25.665876 | controller | -1\n logger.go:42: 16:22:46 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.665880 | controller | -v '^$'\n logger.go:42: 16:22:46 | watcher/4-deploy-with-precreated-account | 2026-03-20 16:26:25.665884 | controller | + APIPOD=\n logger.go:42: 16:22:47 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.665889 | controller | | running command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n 2026-03-20 16:26:25.665893 | controller | \ APIPOD=$(oc get pods -n watcher-kuttl-default -l \"service=watcher-api\" 2026-03-20 16:26:25.665897 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)\n if [ -n \"${APIPOD}\" 2026-03-20 16:26:25.665901 | controller | ]; then\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.665905 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-20 16:26:25.665910 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-20 16:26:25.665916 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-20 16:26:25.665920 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-20 16:26:25.665924 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.665929 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-20 16:26:25.665933 | controller | \ logger.go:42: 16:22:47 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.665937 | controller | watcher-kuttl-default\n logger.go:42: 16:22:47 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.665941 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.665947 | controller | \ logger.go:42: 16:22:47 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.665959 | controller | -v '^$'\n logger.go:42: 16:22:47 | watcher/4-deploy-with-precreated-account | 2026-03-20 16:26:25.665964 | controller | ++ head -1\n logger.go:42: 16:22:47 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.665968 | controller | | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.665972 | controller | \ logger.go:42: 16:22:48 | watcher/4-deploy-with-precreated-account | + APIPOD=\n 2026-03-20 16:26:25.665976 | controller | \ logger.go:42: 16:22:49 | watcher/4-deploy-with-precreated-account | running 2026-03-20 16:26:25.665981 | controller | command: [sh -c set -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc 2026-03-20 16:26:25.665985 | controller | get pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep 2026-03-20 16:26:25.665989 | controller | -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh 2026-03-20 16:26:25.665993 | controller | -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.665997 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-20 16:26:25.666001 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-20 16:26:25.666006 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-20 16:26:25.666010 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-20 16:26:25.666014 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.666018 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-20 16:26:25.666022 | controller | \ logger.go:42: 16:22:49 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.666027 | controller | watcher-kuttl-default\n logger.go:42: 16:22:49 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.666031 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.666035 | controller | \ logger.go:42: 16:22:49 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-20 16:26:25.666039 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.666043 | controller | \ logger.go:42: 16:22:49 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-20 16:26:25.666047 | controller | -1\n logger.go:42: 16:22:49 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.666051 | controller | -v '^$'\n logger.go:42: 16:22:49 | watcher/4-deploy-with-precreated-account | 2026-03-20 16:26:25.666057 | controller | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 16:22:49 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.666061 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 16:22:49 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.666066 | controller | | ++ grep -c '^# Global config'\n logger.go:42: 16:22:49 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.666070 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n 2026-03-20 16:26:25.666074 | controller | \ logger.go:42: 16:22:49 | watcher/4-deploy-with-precreated-account | error: unable 2026-03-20 16:26:25.666078 | controller | to upgrade connection: container not found (\"watcher-api\")\n logger.go:42: 2026-03-20 16:26:25.666082 | controller | 16:22:49 | watcher/4-deploy-with-precreated-account | ++ echo\n logger.go:42: 2026-03-20 16:26:25.666086 | controller | 16:22:49 | watcher/4-deploy-with-precreated-account | + '[' 0 == 1 ']'\n logger.go:42: 2026-03-20 16:26:25.666092 | controller | 16:22:50 | watcher/4-deploy-with-precreated-account | running command: [sh -c set 2026-03-20 16:26:25.666105 | controller | -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get 2026-03-20 16:26:25.666110 | controller | pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep 2026-03-20 16:26:25.666114 | controller | -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh 2026-03-20 16:26:25.666118 | controller | -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.666122 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-20 16:26:25.666126 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-20 16:26:25.666130 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-20 16:26:25.666134 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-20 16:26:25.666144 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.666149 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-20 16:26:25.666153 | controller | \ logger.go:42: 16:22:50 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.666157 | controller | watcher-kuttl-default\n logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.666161 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.666166 | controller | \ logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-20 16:26:25.666170 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.666176 | controller | \ logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-20 16:26:25.666180 | controller | -1\n logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.666184 | controller | -v '^$'\n logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account | 2026-03-20 16:26:25.666189 | controller | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.666193 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.666197 | controller | | ++ grep -c '^# Global config'\n logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.666201 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n 2026-03-20 16:26:25.666205 | controller | \ logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.666209 | controller | '#' Global config\n logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.666214 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.666218 | controller | | ++ grep -c '^# Service config'\n logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.666222 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n 2026-03-20 16:26:25.666226 | controller | \ logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.666230 | controller | '#' Service config\n logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.666234 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.666241 | controller | | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n 2026-03-20 16:26:25.666246 | controller | \ logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account | +++ oc rsh 2026-03-20 16:26:25.666250 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-20 16:26:25.666254 | controller | \ logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.666258 | controller | '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.666262 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.666267 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.666271 | controller | = '[database]' connection = 'mysql+pymysql://watcher_test:4341713c0331ec810a387bb85d10e1d1@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-20 16:26:25.666277 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-20 16:26:25.666281 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.666285 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.666289 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.666293 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.666298 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.666302 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.666306 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.666310 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.666321 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-20 16:26:25.666326 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-20 16:26:25.666330 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-20 16:26:25.666334 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-20 16:26:25.666338 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-20 16:26:25.666343 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.666347 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.666351 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.666355 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-20 16:26:25.666359 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.666363 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' 2026-03-20 16:26:25.666368 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-20 16:26:25.666372 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.666376 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-20 16:26:25.666382 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-20 16:26:25.666386 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.666390 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-20 16:26:25.666394 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-20 16:26:25.666399 | controller | period = 900\n logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.666403 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.666407 | controller | | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.666411 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n 2026-03-20 16:26:25.666415 | controller | \ logger.go:42: 16:22:52 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.666419 | controller | '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot 2026-03-20 16:26:25.666428 | controller | DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 2026-03-20 16:26:25.666432 | controller | 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-20 16:26:25.666436 | controller | '' Options -Indexes +FollowSymLinks +MultiViews 2026-03-20 16:26:25.666442 | controller | AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout 2026-03-20 16:26:25.666446 | controller | ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout 2026-03-20 16:26:25.666450 | controller | proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI 2026-03-20 16:26:25.666455 | controller | configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal 2026-03-20 16:26:25.666459 | controller | group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias 2026-03-20 16:26:25.666463 | controller | / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc 2026-03-20 16:26:25.666468 | controller | configuration '' ServerName watcher-public.watcher-kuttl-default.svc 2026-03-20 16:26:25.666473 | controller | '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the 2026-03-20 16:26:25.666477 | controller | watcher-api TimeOut 80 '##' Directories, there should at least be a declaration 2026-03-20 16:26:25.666481 | controller | for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks 2026-03-20 16:26:25.666485 | controller | +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog 2026-03-20 16:26:25.666489 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' 2026-03-20 16:26:25.666494 | controller | CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel 2026-03-20 16:26:25.666498 | controller | debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess 2026-03-20 16:26:25.666502 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-20 16:26:25.666512 | controller | public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 2026-03-20 16:26:25.666516 | controller | 16:22:52 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-20 16:26:25.666520 | controller | 16:22:53 | watcher/4-deploy-with-precreated-account | running command: [sh -c set 2026-03-20 16:26:25.666528 | controller | -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get 2026-03-20 16:26:25.666533 | controller | pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep 2026-03-20 16:26:25.666537 | controller | -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh 2026-03-20 16:26:25.666541 | controller | -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.666545 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-20 16:26:25.666549 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-20 16:26:25.666553 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-20 16:26:25.666558 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-20 16:26:25.666562 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.666566 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-20 16:26:25.666570 | controller | \ logger.go:42: 16:22:53 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.666574 | controller | watcher-kuttl-default\n logger.go:42: 16:22:53 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.666578 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.666583 | controller | \ logger.go:42: 16:22:53 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-20 16:26:25.666587 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.666591 | controller | \ logger.go:42: 16:22:53 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-20 16:26:25.666595 | controller | -1\n logger.go:42: 16:22:53 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.666599 | controller | -v '^$'\n logger.go:42: 16:22:53 | watcher/4-deploy-with-precreated-account | 2026-03-20 16:26:25.666603 | controller | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 16:22:53 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.666608 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 16:22:53 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.666612 | controller | | ++ grep -c '^# Global config'\n logger.go:42: 16:22:53 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.666616 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n 2026-03-20 16:26:25.666620 | controller | \ logger.go:42: 16:22:53 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.666624 | controller | '#' Global config\n logger.go:42: 16:22:53 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.666628 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:22:53 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.666633 | controller | | ++ grep -c '^# Service config'\n logger.go:42: 16:22:53 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.666637 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n 2026-03-20 16:26:25.666641 | controller | \ logger.go:42: 16:22:54 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.666645 | controller | '#' Service config\n logger.go:42: 16:22:54 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.666649 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:22:54 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.666653 | controller | | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n 2026-03-20 16:26:25.666660 | controller | \ logger.go:42: 16:22:54 | watcher/4-deploy-with-precreated-account | +++ oc rsh 2026-03-20 16:26:25.666664 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-20 16:26:25.666669 | controller | \ logger.go:42: 16:22:54 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.666673 | controller | '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.666683 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.666687 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.666702 | controller | = '[database]' connection = 'mysql+pymysql://watcher_test:4341713c0331ec810a387bb85d10e1d1@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-20 16:26:25.666712 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-20 16:26:25.666716 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.666720 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.666725 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.666729 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.666733 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.666737 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.666742 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.666746 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.666750 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-20 16:26:25.666754 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-20 16:26:25.666758 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-20 16:26:25.666762 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-20 16:26:25.666767 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-20 16:26:25.666771 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.666775 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.666779 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.666783 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-20 16:26:25.666788 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.666792 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' 2026-03-20 16:26:25.666796 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-20 16:26:25.666802 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.666806 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-20 16:26:25.666810 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-20 16:26:25.666814 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.666820 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-20 16:26:25.666824 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-20 16:26:25.666829 | controller | period = 900\n logger.go:42: 16:22:54 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.666833 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:22:54 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.666837 | controller | | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 16:22:54 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.666841 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n 2026-03-20 16:26:25.666845 | controller | \ logger.go:42: 16:22:54 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.666849 | controller | '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot 2026-03-20 16:26:25.666858 | controller | DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 2026-03-20 16:26:25.666862 | controller | 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-20 16:26:25.666866 | controller | '' Options -Indexes +FollowSymLinks +MultiViews 2026-03-20 16:26:25.666870 | controller | AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout 2026-03-20 16:26:25.666874 | controller | ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout 2026-03-20 16:26:25.666885 | controller | proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI 2026-03-20 16:26:25.666890 | controller | configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal 2026-03-20 16:26:25.666894 | controller | group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias 2026-03-20 16:26:25.666899 | controller | / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc 2026-03-20 16:26:25.666903 | controller | configuration '' ServerName watcher-public.watcher-kuttl-default.svc 2026-03-20 16:26:25.666907 | controller | '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the 2026-03-20 16:26:25.666911 | controller | watcher-api TimeOut 80 '##' Directories, there should at least be a declaration 2026-03-20 16:26:25.666916 | controller | for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks 2026-03-20 16:26:25.666920 | controller | +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog 2026-03-20 16:26:25.666924 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' 2026-03-20 16:26:25.666928 | controller | CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel 2026-03-20 16:26:25.666932 | controller | debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess 2026-03-20 16:26:25.666936 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-20 16:26:25.666941 | controller | public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 2026-03-20 16:26:25.666945 | controller | 16:22:54 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-20 16:26:25.666949 | controller | 16:22:55 | watcher/4-deploy-with-precreated-account | running command: [sh -c set 2026-03-20 16:26:25.666953 | controller | -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get 2026-03-20 16:26:25.666957 | controller | pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep 2026-03-20 16:26:25.666972 | controller | -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh 2026-03-20 16:26:25.666977 | controller | -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.666981 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-20 16:26:25.666987 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-20 16:26:25.666991 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-20 16:26:25.666995 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-20 16:26:25.667000 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.667004 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-20 16:26:25.667008 | controller | \ logger.go:42: 16:22:55 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.667012 | controller | watcher-kuttl-default\n logger.go:42: 16:22:55 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.667016 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.667021 | controller | \ logger.go:42: 16:22:55 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-20 16:26:25.667025 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.667029 | controller | \ logger.go:42: 16:22:55 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-20 16:26:25.667033 | controller | -1\n logger.go:42: 16:22:55 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.667045 | controller | -v '^$'\n logger.go:42: 16:22:56 | watcher/4-deploy-with-precreated-account | 2026-03-20 16:26:25.667049 | controller | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 16:22:56 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.667053 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 16:22:56 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.667057 | controller | | ++ grep -c '^# Global config'\n logger.go:42: 16:22:56 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.667062 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n 2026-03-20 16:26:25.667066 | controller | \ logger.go:42: 16:22:56 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.667070 | controller | '#' Global config\n logger.go:42: 16:22:56 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.667074 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:22:56 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.667085 | controller | | ++ grep -c '^# Service config'\n logger.go:42: 16:22:56 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.667090 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n 2026-03-20 16:26:25.667094 | controller | \ logger.go:42: 16:22:56 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.667098 | controller | '#' Service config\n logger.go:42: 16:22:56 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.667102 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:22:56 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.667106 | controller | | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n 2026-03-20 16:26:25.667111 | controller | \ logger.go:42: 16:22:56 | watcher/4-deploy-with-precreated-account | +++ oc rsh 2026-03-20 16:26:25.667115 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-20 16:26:25.667121 | controller | \ logger.go:42: 16:22:56 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.667125 | controller | '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.667136 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.667141 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.667145 | controller | = '[database]' connection = 'mysql+pymysql://watcher_test:4341713c0331ec810a387bb85d10e1d1@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-20 16:26:25.667152 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-20 16:26:25.667156 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.667160 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.667165 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.667178 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.667182 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.667186 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.667191 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.667195 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.667199 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-20 16:26:25.667203 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-20 16:26:25.667207 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-20 16:26:25.667212 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-20 16:26:25.667216 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-20 16:26:25.667220 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.667224 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.667229 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.667233 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-20 16:26:25.667237 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.667241 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' 2026-03-20 16:26:25.667245 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-20 16:26:25.667250 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.667254 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-20 16:26:25.667258 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-20 16:26:25.667262 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.667266 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-20 16:26:25.667279 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-20 16:26:25.667283 | controller | period = 900\n logger.go:42: 16:22:56 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.667287 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:22:56 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.667291 | controller | | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 16:22:56 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.667296 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n 2026-03-20 16:26:25.667307 | controller | \ logger.go:42: 16:22:56 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.667312 | controller | '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot 2026-03-20 16:26:25.667320 | controller | DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 2026-03-20 16:26:25.667324 | controller | 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-20 16:26:25.667328 | controller | '' Options -Indexes +FollowSymLinks +MultiViews 2026-03-20 16:26:25.667332 | controller | AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout 2026-03-20 16:26:25.667337 | controller | ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout 2026-03-20 16:26:25.667341 | controller | proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI 2026-03-20 16:26:25.667345 | controller | configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal 2026-03-20 16:26:25.667349 | controller | group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias 2026-03-20 16:26:25.667353 | controller | / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc 2026-03-20 16:26:25.667357 | controller | configuration '' ServerName watcher-public.watcher-kuttl-default.svc 2026-03-20 16:26:25.667362 | controller | '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the 2026-03-20 16:26:25.667366 | controller | watcher-api TimeOut 80 '##' Directories, there should at least be a declaration 2026-03-20 16:26:25.667370 | controller | for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks 2026-03-20 16:26:25.667374 | controller | +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog 2026-03-20 16:26:25.667378 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' 2026-03-20 16:26:25.667382 | controller | CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel 2026-03-20 16:26:25.667386 | controller | debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess 2026-03-20 16:26:25.667390 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-20 16:26:25.667394 | controller | public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 2026-03-20 16:26:25.667399 | controller | 16:22:56 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-20 16:26:25.667403 | controller | 16:22:57 | watcher/4-deploy-with-precreated-account | running command: [sh -c set 2026-03-20 16:26:25.667407 | controller | -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get 2026-03-20 16:26:25.667411 | controller | pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep 2026-03-20 16:26:25.667415 | controller | -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh 2026-03-20 16:26:25.667419 | controller | -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.667425 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-20 16:26:25.667430 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-20 16:26:25.667434 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-20 16:26:25.667438 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-20 16:26:25.667442 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.667446 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-20 16:26:25.667450 | controller | \ logger.go:42: 16:22:57 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.667455 | controller | watcher-kuttl-default\n logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.667459 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.667470 | controller | \ logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-20 16:26:25.667475 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.667479 | controller | \ logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-20 16:26:25.667483 | controller | -1\n logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.667487 | controller | -v '^$'\n logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account | 2026-03-20 16:26:25.667491 | controller | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.667496 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.667500 | controller | | ++ grep -c '^# Global config'\n logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.667504 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n 2026-03-20 16:26:25.667508 | controller | \ logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.667512 | controller | '#' Global config\n logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.667517 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.667521 | controller | | ++ grep -c '^# Service config'\n logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.667527 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n 2026-03-20 16:26:25.667531 | controller | \ logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.667544 | controller | '#' Service config\n logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.667548 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.667552 | controller | | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n 2026-03-20 16:26:25.667556 | controller | \ logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account | +++ oc rsh 2026-03-20 16:26:25.667576 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-20 16:26:25.667584 | controller | \ logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.667592 | controller | '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.667597 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.667602 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.667606 | controller | = '[database]' connection = 'mysql+pymysql://watcher_test:4341713c0331ec810a387bb85d10e1d1@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-20 16:26:25.667615 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-20 16:26:25.667620 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.667624 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.667629 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.667634 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.667639 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.667644 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.667649 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.667654 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.667659 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-20 16:26:25.667663 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-20 16:26:25.667668 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-20 16:26:25.667673 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-20 16:26:25.667678 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-20 16:26:25.667683 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.667709 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.667718 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.667724 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-20 16:26:25.667729 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.667735 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' 2026-03-20 16:26:25.667739 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-20 16:26:25.667744 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.667748 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-20 16:26:25.667752 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-20 16:26:25.667756 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.667760 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-20 16:26:25.667765 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-20 16:26:25.667771 | controller | period = 900\n logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.667775 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.667779 | controller | | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.667783 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n 2026-03-20 16:26:25.667788 | controller | \ logger.go:42: 16:22:59 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.667792 | controller | '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot 2026-03-20 16:26:25.667800 | controller | DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 2026-03-20 16:26:25.667804 | controller | 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-20 16:26:25.667809 | controller | '' Options -Indexes +FollowSymLinks +MultiViews 2026-03-20 16:26:25.667813 | controller | AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout 2026-03-20 16:26:25.667817 | controller | ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout 2026-03-20 16:26:25.667821 | controller | proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI 2026-03-20 16:26:25.667825 | controller | configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal 2026-03-20 16:26:25.667829 | controller | group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias 2026-03-20 16:26:25.667834 | controller | / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc 2026-03-20 16:26:25.667838 | controller | configuration '' ServerName watcher-public.watcher-kuttl-default.svc 2026-03-20 16:26:25.667842 | controller | '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the 2026-03-20 16:26:25.667846 | controller | watcher-api TimeOut 80 '##' Directories, there should at least be a declaration 2026-03-20 16:26:25.667850 | controller | for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks 2026-03-20 16:26:25.667854 | controller | +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog 2026-03-20 16:26:25.667858 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' 2026-03-20 16:26:25.667863 | controller | CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel 2026-03-20 16:26:25.667868 | controller | debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess 2026-03-20 16:26:25.667873 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-20 16:26:25.667878 | controller | public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 2026-03-20 16:26:25.667884 | controller | 16:22:59 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-20 16:26:25.667890 | controller | 16:23:00 | watcher/4-deploy-with-precreated-account | running command: [sh -c set 2026-03-20 16:26:25.667896 | controller | -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get 2026-03-20 16:26:25.667901 | controller | pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep 2026-03-20 16:26:25.667914 | controller | -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh 2026-03-20 16:26:25.667918 | controller | -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.667922 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-20 16:26:25.667927 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-20 16:26:25.667934 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-20 16:26:25.667941 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-20 16:26:25.667945 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.667949 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-20 16:26:25.667953 | controller | \ logger.go:42: 16:23:00 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.667958 | controller | watcher-kuttl-default\n logger.go:42: 16:23:00 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.667962 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.667966 | controller | \ logger.go:42: 16:23:00 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-20 16:26:25.667970 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.667974 | controller | \ logger.go:42: 16:23:00 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.667979 | controller | -v '^$'\n logger.go:42: 16:23:00 | watcher/4-deploy-with-precreated-account | 2026-03-20 16:26:25.667983 | controller | ++ head -1\n logger.go:42: 16:23:00 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.667990 | controller | | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 16:23:00 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.667995 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 16:23:00 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.668001 | controller | | ++ grep -c '^# Global config'\n logger.go:42: 16:23:00 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.668007 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n 2026-03-20 16:26:25.668012 | controller | \ logger.go:42: 16:23:00 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.668018 | controller | '#' Global config\n logger.go:42: 16:23:00 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.668023 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:23:00 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.668027 | controller | | ++ grep -c '^# Service config'\n logger.go:42: 16:23:00 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.668031 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n 2026-03-20 16:26:25.668035 | controller | \ logger.go:42: 16:23:00 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.668040 | controller | '#' Service config\n logger.go:42: 16:23:00 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.668044 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:23:00 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.668050 | controller | | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n 2026-03-20 16:26:25.668056 | controller | \ logger.go:42: 16:23:00 | watcher/4-deploy-with-precreated-account | +++ oc rsh 2026-03-20 16:26:25.668061 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-20 16:26:25.668067 | controller | \ logger.go:42: 16:23:01 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.668082 | controller | '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.668089 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.668094 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.668098 | controller | = '[database]' connection = 'mysql+pymysql://watcher_test:4341713c0331ec810a387bb85d10e1d1@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-20 16:26:25.668104 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-20 16:26:25.668108 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.668120 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.668124 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.668129 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.668134 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.668138 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.668143 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.668147 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.668151 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-20 16:26:25.668155 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-20 16:26:25.668159 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-20 16:26:25.668163 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-20 16:26:25.668168 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-20 16:26:25.668172 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.668176 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.668181 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.668186 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-20 16:26:25.668192 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.668197 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' 2026-03-20 16:26:25.668203 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-20 16:26:25.668207 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.668212 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-20 16:26:25.668216 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-20 16:26:25.668220 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.668224 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-20 16:26:25.668229 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-20 16:26:25.668235 | controller | period = 900\n logger.go:42: 16:23:01 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.668240 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:23:01 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.668248 | controller | | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 16:23:01 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.668252 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n 2026-03-20 16:26:25.668257 | controller | \ logger.go:42: 16:23:01 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.668261 | controller | '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot 2026-03-20 16:26:25.668269 | controller | DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 2026-03-20 16:26:25.668273 | controller | 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-20 16:26:25.668277 | controller | '' Options -Indexes +FollowSymLinks +MultiViews 2026-03-20 16:26:25.668281 | controller | AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout 2026-03-20 16:26:25.668286 | controller | ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout 2026-03-20 16:26:25.668290 | controller | proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI 2026-03-20 16:26:25.668294 | controller | configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal 2026-03-20 16:26:25.668298 | controller | group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias 2026-03-20 16:26:25.668302 | controller | / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc 2026-03-20 16:26:25.668307 | controller | configuration '' ServerName watcher-public.watcher-kuttl-default.svc 2026-03-20 16:26:25.668311 | controller | '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the 2026-03-20 16:26:25.668323 | controller | watcher-api TimeOut 80 '##' Directories, there should at least be a declaration 2026-03-20 16:26:25.668329 | controller | for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks 2026-03-20 16:26:25.668334 | controller | +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog 2026-03-20 16:26:25.668340 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' 2026-03-20 16:26:25.668345 | controller | CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel 2026-03-20 16:26:25.668349 | controller | debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess 2026-03-20 16:26:25.668353 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-20 16:26:25.668358 | controller | public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 2026-03-20 16:26:25.668364 | controller | 16:23:01 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-20 16:26:25.668368 | controller | 16:23:02 | watcher/4-deploy-with-precreated-account | running command: [sh -c set 2026-03-20 16:26:25.668373 | controller | -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get 2026-03-20 16:26:25.668379 | controller | pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep 2026-03-20 16:26:25.668384 | controller | -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh 2026-03-20 16:26:25.668390 | controller | -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.668394 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-20 16:26:25.668398 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-20 16:26:25.668403 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-20 16:26:25.668407 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-20 16:26:25.668413 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.668418 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-20 16:26:25.668422 | controller | \ logger.go:42: 16:23:02 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.668426 | controller | watcher-kuttl-default\n logger.go:42: 16:23:02 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.668430 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.668434 | controller | \ logger.go:42: 16:23:02 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.668439 | controller | -v '^$'\n logger.go:42: 16:23:02 | watcher/4-deploy-with-precreated-account | 2026-03-20 16:26:25.668443 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.668447 | controller | \ logger.go:42: 16:23:02 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-20 16:26:25.668451 | controller | -1\n logger.go:42: 16:23:02 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0\n 2026-03-20 16:26:25.668455 | controller | \ logger.go:42: 16:23:02 | watcher/4-deploy-with-precreated-account | + '[' -n 2026-03-20 16:26:25.668459 | controller | watcher-kuttl-api-0 ']'\n logger.go:42: 16:23:02 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.668464 | controller | | ++ grep -c '^# Global config'\n logger.go:42: 16:23:02 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.668468 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n 2026-03-20 16:26:25.668472 | controller | \ logger.go:42: 16:23:02 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.668478 | controller | '#' Global config\n logger.go:42: 16:23:02 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.668483 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:23:02 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.668488 | controller | | ++ grep -c '^# Service config'\n logger.go:42: 16:23:02 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.668493 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n 2026-03-20 16:26:25.668499 | controller | \ logger.go:42: 16:23:03 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.668505 | controller | '#' Service config\n logger.go:42: 16:23:03 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.668549 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:23:03 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.668566 | controller | | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n 2026-03-20 16:26:25.668572 | controller | \ logger.go:42: 16:23:03 | watcher/4-deploy-with-precreated-account | +++ oc rsh 2026-03-20 16:26:25.668576 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-20 16:26:25.668581 | controller | \ logger.go:42: 16:23:03 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.668587 | controller | '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.668592 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.668607 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.668617 | controller | = '[database]' connection = 'mysql+pymysql://watcher_test:4341713c0331ec810a387bb85d10e1d1@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-20 16:26:25.668623 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-20 16:26:25.668628 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.668632 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.668636 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.668640 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.668644 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.668649 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.668653 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.668657 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.668661 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-20 16:26:25.668665 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-20 16:26:25.668733 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-20 16:26:25.668738 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-20 16:26:25.668742 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-20 16:26:25.668774 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.668781 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.668787 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.668792 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-20 16:26:25.668797 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.668803 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' 2026-03-20 16:26:25.668808 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-20 16:26:25.668813 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.668817 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-20 16:26:25.668821 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-20 16:26:25.668826 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.668830 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-20 16:26:25.668834 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-20 16:26:25.668838 | controller | period = 900\n logger.go:42: 16:23:03 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.668851 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:23:03 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.668855 | controller | | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 16:23:03 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.668859 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n 2026-03-20 16:26:25.668867 | controller | \ logger.go:42: 16:23:03 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.668880 | controller | '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot 2026-03-20 16:26:25.668889 | controller | DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 2026-03-20 16:26:25.668893 | controller | 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-20 16:26:25.668897 | controller | '' Options -Indexes +FollowSymLinks +MultiViews 2026-03-20 16:26:25.668901 | controller | AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout 2026-03-20 16:26:25.668905 | controller | ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout 2026-03-20 16:26:25.668909 | controller | proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI 2026-03-20 16:26:25.668914 | controller | configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal 2026-03-20 16:26:25.668918 | controller | group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias 2026-03-20 16:26:25.668922 | controller | / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc 2026-03-20 16:26:25.668926 | controller | configuration '' ServerName watcher-public.watcher-kuttl-default.svc 2026-03-20 16:26:25.668930 | controller | '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the 2026-03-20 16:26:25.668934 | controller | watcher-api TimeOut 80 '##' Directories, there should at least be a declaration 2026-03-20 16:26:25.668939 | controller | for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks 2026-03-20 16:26:25.668948 | controller | +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog 2026-03-20 16:26:25.668952 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' 2026-03-20 16:26:25.668957 | controller | CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel 2026-03-20 16:26:25.668961 | controller | debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess 2026-03-20 16:26:25.668965 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-20 16:26:25.668969 | controller | public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 2026-03-20 16:26:25.668973 | controller | 16:23:03 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-20 16:26:25.668977 | controller | 16:23:04 | watcher/4-deploy-with-precreated-account | running command: [sh -c set 2026-03-20 16:26:25.668981 | controller | -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get 2026-03-20 16:26:25.668986 | controller | pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep 2026-03-20 16:26:25.668990 | controller | -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh 2026-03-20 16:26:25.668994 | controller | -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.668998 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-20 16:26:25.669002 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-20 16:26:25.669006 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-20 16:26:25.669010 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-20 16:26:25.669024 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.669029 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-20 16:26:25.669033 | controller | \ logger.go:42: 16:23:04 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.669037 | controller | watcher-kuttl-default\n logger.go:42: 16:23:04 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.669041 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.669045 | controller | \ logger.go:42: 16:23:04 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-20 16:26:25.669050 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.669054 | controller | \ logger.go:42: 16:23:04 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-20 16:26:25.669058 | controller | -1\n logger.go:42: 16:23:04 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.669062 | controller | -v '^$'\n logger.go:42: 16:23:04 | watcher/4-deploy-with-precreated-account | 2026-03-20 16:26:25.669066 | controller | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 16:23:04 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.669078 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 16:23:04 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.669083 | controller | | ++ grep -c '^# Global config'\n logger.go:42: 16:23:04 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.669087 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n 2026-03-20 16:26:25.669091 | controller | \ logger.go:42: 16:23:05 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.669095 | controller | '#' Global config\n logger.go:42: 16:23:05 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.669100 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:23:05 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.669104 | controller | | ++ grep -c '^# Service config'\n logger.go:42: 16:23:05 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.669108 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n 2026-03-20 16:26:25.669112 | controller | \ logger.go:42: 16:23:05 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.669116 | controller | '#' Service config\n logger.go:42: 16:23:05 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.669120 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:23:05 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.669125 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-20 16:26:25.669129 | controller | \ logger.go:42: 16:23:05 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.669133 | controller | -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n 2026-03-20 16:26:25.669137 | controller | \ logger.go:42: 16:23:05 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.669142 | controller | '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.669148 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.669152 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.669157 | controller | = '[database]' connection = 'mysql+pymysql://watcher_test:4341713c0331ec810a387bb85d10e1d1@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-20 16:26:25.669165 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-20 16:26:25.669169 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.669174 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.669178 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.669182 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.669186 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.669190 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.669196 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.669211 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.669217 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-20 16:26:25.669222 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-20 16:26:25.669226 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-20 16:26:25.669230 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-20 16:26:25.669234 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-20 16:26:25.669239 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.669243 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.669247 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.669251 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-20 16:26:25.669255 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.669260 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' 2026-03-20 16:26:25.669264 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-20 16:26:25.669276 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.669283 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-20 16:26:25.669287 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-20 16:26:25.669292 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.669296 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-20 16:26:25.669300 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-20 16:26:25.669304 | controller | period = 900\n logger.go:42: 16:23:05 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.669308 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:23:05 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.669312 | controller | | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 16:23:05 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.669317 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n 2026-03-20 16:26:25.669321 | controller | \ logger.go:42: 16:23:05 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.669325 | controller | '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot 2026-03-20 16:26:25.669335 | controller | DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 2026-03-20 16:26:25.669340 | controller | 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-20 16:26:25.669344 | controller | '' Options -Indexes +FollowSymLinks +MultiViews 2026-03-20 16:26:25.669348 | controller | AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout 2026-03-20 16:26:25.669352 | controller | ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout 2026-03-20 16:26:25.669357 | controller | proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI 2026-03-20 16:26:25.669361 | controller | configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal 2026-03-20 16:26:25.669365 | controller | group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias 2026-03-20 16:26:25.669369 | controller | / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc 2026-03-20 16:26:25.669374 | controller | configuration '' ServerName watcher-public.watcher-kuttl-default.svc 2026-03-20 16:26:25.669380 | controller | '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the 2026-03-20 16:26:25.669385 | controller | watcher-api TimeOut 80 '##' Directories, there should at least be a declaration 2026-03-20 16:26:25.669390 | controller | for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks 2026-03-20 16:26:25.669396 | controller | +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog 2026-03-20 16:26:25.669401 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' 2026-03-20 16:26:25.669405 | controller | CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel 2026-03-20 16:26:25.669409 | controller | debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess 2026-03-20 16:26:25.669414 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-20 16:26:25.669418 | controller | public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 2026-03-20 16:26:25.669422 | controller | 16:23:05 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-20 16:26:25.669426 | controller | 16:23:07 | watcher/4-deploy-with-precreated-account | running command: [sh -c set 2026-03-20 16:26:25.669430 | controller | -euxo pipefail\n oc project watcher-kuttl-default\n APIPOD=$(oc get 2026-03-20 16:26:25.669434 | controller | pods -n watcher-kuttl-default -l \"service=watcher-api\" -ocustom-columns=:metadata.name|grep 2026-03-20 16:26:25.669439 | controller | -v ^$|head -1)\n if [ -n \"${APIPOD}\" ]; then\n [ $(echo $(oc rsh 2026-03-20 16:26:25.669443 | controller | -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.669447 | controller | |grep -c \"^# Global config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api 2026-03-20 16:26:25.669451 | controller | ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) |grep -c \"^# 2026-03-20 16:26:25.669455 | controller | Service config\") == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat 2026-03-20 16:26:25.669470 | controller | /etc/watcher/watcher.conf.d/00-default.conf) |grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem') 2026-03-20 16:26:25.669475 | controller | == 1 ]\n [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.669479 | controller | |grep -czPo 'TimeOut 80') == 1 ]\n else\n exit 1\n fi\n ]\n 2026-03-20 16:26:25.669487 | controller | \ logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.669500 | controller | watcher-kuttl-default\n logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.669506 | controller | | Already on project \"watcher-kuttl-default\" on server \"https://api.crc.testing:6443\".\n 2026-03-20 16:26:25.669511 | controller | \ logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-20 16:26:25.669517 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name\n 2026-03-20 16:26:25.669522 | controller | \ logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-20 16:26:25.669526 | controller | -1\n logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.669530 | controller | -v '^$'\n logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account | 2026-03-20 16:26:25.669534 | controller | + APIPOD=watcher-kuttl-api-0\n logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.669538 | controller | | + '[' -n watcher-kuttl-api-0 ']'\n logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.669543 | controller | | ++ grep -c '^# Global config'\n logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.669547 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf\n 2026-03-20 16:26:25.669551 | controller | \ logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.669555 | controller | '#' Global config\n logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.669559 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.669564 | controller | | ++ grep -c '^# Service config'\n logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.669568 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf\n 2026-03-20 16:26:25.669572 | controller | \ logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.669578 | controller | '#' Service config\n logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.669583 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.669589 | controller | | ++ grep -czPo '\\[prometheus_client\\]\\s+host\\s+=\\s+metric-storage-prometheus.watcher-kuttl-default.svc\\s+port\\s+=\\s+9090\\s+cafile\\s+=\\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'\n 2026-03-20 16:26:25.669594 | controller | \ logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account | +++ oc rsh 2026-03-20 16:26:25.669598 | controller | -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf\n 2026-03-20 16:26:25.669603 | controller | \ logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.669607 | controller | '[DEFAULT]' state_path = /var/lib/watcher transport_url = 'rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1' 2026-03-20 16:26:25.669611 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.669615 | controller | '#' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.669619 | controller | = '[database]' connection = 'mysql+pymysql://watcher_test:4341713c0331ec810a387bb85d10e1d1@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf' 2026-03-20 16:26:25.669626 | controller | '[oslo_policy]' policy_file = /etc/watcher/policy.yaml.sample '[oslo_messaging_notifications]' 2026-03-20 16:26:25.669631 | controller | driver = noop '[oslo_messaging_rabbit]' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.669637 | controller | amqp_durable_queues=true '[keystone_authtoken]' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.669641 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.669647 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.669651 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.669656 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.669661 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.669673 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.669678 | controller | '[watcher_clients_auth]' project_domain_name = Default project_name = service user_domain_name 2026-03-20 16:26:25.669682 | controller | = Default password = password username = watcher auth_type = password auth_url = 2026-03-20 16:26:25.669686 | controller | https://keystone-internal.watcher-kuttl-default.svc:5000 interface = internal cafile 2026-03-20 16:26:25.669690 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem '[oslo_concurrency]' lock_path 2026-03-20 16:26:25.669710 | controller | = /var/lib/watcher/tmp '[watcher_datasources]' datasources = prometheus '[cache]' 2026-03-20 16:26:25.669714 | controller | backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.669718 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.669723 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.669727 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 '[prometheus_client]' 2026-03-20 16:26:25.669731 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.669735 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem '[aetos_client]' 2026-03-20 16:26:25.669739 | controller | endpoint_type = internal region_name = regionOne '[cinder_client]' endpoint_type 2026-03-20 16:26:25.669743 | controller | = internal region_name = regionOne '[glance_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.669747 | controller | = regionOne '[ironic_client]' endpoint_type = internal region_name = regionOne '[keystone_client]' 2026-03-20 16:26:25.669752 | controller | interface = internal region_name = regionOne '[neutron_client]' endpoint_type = 2026-03-20 16:26:25.669756 | controller | internal region_name = regionOne '[nova_client]' endpoint_type = internal region_name 2026-03-20 16:26:25.669760 | controller | = regionOne '[placement_client]' interface = internal region_name = regionOne '[watcher_cluster_data_model_collectors.compute]' 2026-03-20 16:26:25.669766 | controller | period = 900 '[watcher_cluster_data_model_collectors.baremetal]' period = 900 '[watcher_cluster_data_model_collectors.storage]' 2026-03-20 16:26:25.669771 | controller | period = 900\n logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.669775 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.669779 | controller | | ++ grep -czPo 'TimeOut 80'\n logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account 2026-03-20 16:26:25.669783 | controller | | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf\n 2026-03-20 16:26:25.669788 | controller | \ logger.go:42: 16:23:08 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.669792 | controller | '#' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '' ServerName watcher-internal.watcher-kuttl-default.svc '##' Vhost docroot 2026-03-20 16:26:25.669803 | controller | DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the watcher-api TimeOut 2026-03-20 16:26:25.669809 | controller | 80 '##' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-20 16:26:25.669815 | controller | '' Options -Indexes +FollowSymLinks +MultiViews 2026-03-20 16:26:25.669821 | controller | AllowOverride None Require all granted '' '##' Logging ErrorLog /dev/stdout 2026-03-20 16:26:25.669826 | controller | ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' CustomLog /dev/stdout 2026-03-20 16:26:25.669832 | controller | proxy env=forwarded '##' set watcher log level to debug LogLevel debug '##' WSGI 2026-03-20 16:26:25.669837 | controller | configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess internal display-name=internal 2026-03-20 16:26:25.669841 | controller | group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup internal WSGIScriptAlias 2026-03-20 16:26:25.669845 | controller | / '\"/usr/bin/watcher-api-wsgi\"' '' '#' public vhost watcher-public.watcher-kuttl-default.svc 2026-03-20 16:26:25.669850 | controller | configuration '' ServerName watcher-public.watcher-kuttl-default.svc 2026-03-20 16:26:25.669854 | controller | '##' Vhost docroot DocumentRoot '\"/var/www/cgi-bin\"' '#' Set the timeout for the 2026-03-20 16:26:25.669858 | controller | watcher-api TimeOut 80 '##' Directories, there should at least be a declaration 2026-03-20 16:26:25.669862 | controller | for /var/www/cgi-bin '' Options -Indexes +FollowSymLinks 2026-03-20 16:26:25.669866 | controller | +MultiViews AllowOverride None Require all granted '' '##' Logging ErrorLog 2026-03-20 16:26:25.669870 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined 'env=!forwarded' 2026-03-20 16:26:25.669874 | controller | CustomLog /dev/stdout proxy env=forwarded '##' set watcher log level to debug LogLevel 2026-03-20 16:26:25.669879 | controller | debug '##' WSGI configuration WSGIApplicationGroup '%{GLOBAL}' WSGIDaemonProcess 2026-03-20 16:26:25.669891 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-20 16:26:25.669895 | controller | public WSGIScriptAlias / '\"/usr/bin/watcher-api-wsgi\"' ''\n logger.go:42: 2026-03-20 16:26:25.669899 | controller | 16:23:08 | watcher/4-deploy-with-precreated-account | + '[' 1 == 1 ']'\n logger.go:42: 2026-03-20 16:26:25.669904 | controller | 16:23:08 | watcher/4-deploy-with-precreated-account | test step completed 4-deploy-with-precreated-account\n 2026-03-20 16:26:25.669908 | controller | \ logger.go:42: 16:23:08 | watcher/5-cleanup-watcher | starting test step 5-cleanup-watcher\n 2026-03-20 16:26:25.669912 | controller | \ logger.go:42: 16:23:08 | watcher/5-cleanup-watcher | running command: [sh -c 2026-03-20 16:26:25.669916 | controller | set -ex\n oc exec -n watcher-kuttl-default openstackclient -- openstack service 2026-03-20 16:26:25.669920 | controller | list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]\n ]\n logger.go:42: 2026-03-20 16:26:25.669925 | controller | 16:23:08 | watcher/5-cleanup-watcher | + oc exec -n watcher-kuttl-default openstackclient 2026-03-20 16:26:25.669929 | controller | -- openstack service list -f value -c Name -c Type\n logger.go:42: 16:23:08 | 2026-03-20 16:26:25.669933 | controller | watcher/5-cleanup-watcher | ++ grep -c '^watcher'\n logger.go:42: 16:23:11 | 2026-03-20 16:26:25.669937 | controller | watcher/5-cleanup-watcher | + '[' 1 == 0 ']'\n logger.go:42: 16:23:12 | watcher/5-cleanup-watcher 2026-03-20 16:26:25.669941 | controller | | running command: [sh -c set -ex\n oc exec -n watcher-kuttl-default openstackclient 2026-03-20 16:26:25.669945 | controller | -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 2026-03-20 16:26:25.669950 | controller | ]\n ]\n logger.go:42: 16:23:12 | watcher/5-cleanup-watcher | + oc exec 2026-03-20 16:26:25.669954 | controller | -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-20 16:26:25.669958 | controller | -c Type\n logger.go:42: 16:23:12 | watcher/5-cleanup-watcher | ++ grep -c '^watcher'\n 2026-03-20 16:26:25.669965 | controller | \ logger.go:42: 16:23:14 | watcher/5-cleanup-watcher | + '[' 0 == 0 ']'\n logger.go:42: 2026-03-20 16:26:25.669971 | controller | 16:23:15 | watcher/5-cleanup-watcher | running command: [sh -c set -ex\n oc 2026-03-20 16:26:25.669977 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-20 16:26:25.669982 | controller | -c Name -c Type | [ $(grep -c ^watcher) == 0 ]\n ]\n logger.go:42: 16:23:15 2026-03-20 16:26:25.669988 | controller | | watcher/5-cleanup-watcher | + oc exec -n watcher-kuttl-default openstackclient 2026-03-20 16:26:25.669994 | controller | -- openstack service list -f value -c Name -c Type\n logger.go:42: 16:23:15 | 2026-03-20 16:26:25.669999 | controller | watcher/5-cleanup-watcher | ++ grep -c '^watcher'\n logger.go:42: 16:23:17 | 2026-03-20 16:26:25.670005 | controller | watcher/5-cleanup-watcher | + '[' 0 == 0 ']'\n logger.go:42: 16:23:18 | watcher/5-cleanup-watcher 2026-03-20 16:26:25.670009 | controller | | running command: [sh -c set -ex\n oc exec -n watcher-kuttl-default openstackclient 2026-03-20 16:26:25.670013 | controller | -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 2026-03-20 16:26:25.670017 | controller | ]\n ]\n logger.go:42: 16:23:18 | watcher/5-cleanup-watcher | + oc exec 2026-03-20 16:26:25.670022 | controller | -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-20 16:26:25.670026 | controller | -c Type\n logger.go:42: 16:23:18 | watcher/5-cleanup-watcher | ++ grep -c '^watcher'\n 2026-03-20 16:26:25.670034 | controller | \ logger.go:42: 16:23:21 | watcher/5-cleanup-watcher | + '[' 0 == 0 ']'\n logger.go:42: 2026-03-20 16:26:25.670038 | controller | 16:23:21 | watcher/5-cleanup-watcher | test step completed 5-cleanup-watcher\n logger.go:42: 2026-03-20 16:26:25.670042 | controller | 16:23:21 | watcher | skipping kubernetes event logging\n=== CONT kuttl/harness/watcher-cinder\n 2026-03-20 16:26:25.670047 | controller | \ logger.go:42: 16:23:21 | watcher-cinder | Skipping creation of user-supplied 2026-03-20 16:26:25.670051 | controller | namespace: watcher-kuttl-default\n logger.go:42: 16:23:21 | watcher-cinder/0-cleanup-watcher 2026-03-20 16:26:25.670056 | controller | | starting test step 0-cleanup-watcher\n logger.go:42: 16:23:21 | watcher-cinder/0-cleanup-watcher 2026-03-20 16:26:25.670062 | controller | | test step completed 0-cleanup-watcher\n logger.go:42: 16:23:21 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670067 | controller | | starting test step 1-deploy-watcher-no-cinder\n logger.go:42: 16:23:21 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670073 | controller | | Watcher:watcher-kuttl-default/watcher-kuttl created\n logger.go:42: 16:23:21 2026-03-20 16:26:25.670077 | controller | | watcher-cinder/1-deploy-watcher-no-cinder | running command: [sh -c set -euxo 2026-03-20 16:26:25.670081 | controller | pipefail\n # check that the decision engine correctly detects that there 2026-03-20 16:26:25.670085 | controller | is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.670089 | controller | | grep -c 'Block storage service is not enabled, skipping storage collector')\" 2026-03-20 16:26:25.670101 | controller | == 2 ]\n ]\n logger.go:42: 16:23:21 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670106 | controller | | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.670110 | controller | 16:23:21 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c 'Block storage 2026-03-20 16:26:25.670117 | controller | service is not enabled, skipping storage collector'\n logger.go:42: 16:23:21 2026-03-20 16:26:25.670121 | controller | | watcher-cinder/1-deploy-watcher-no-cinder | error: error from server (NotFound): 2026-03-20 16:26:25.670125 | controller | pods \"watcher-kuttl-decision-engine-0\" not found in namespace \"watcher-kuttl-default\"\n 2026-03-20 16:26:25.670131 | controller | \ logger.go:42: 16:23:21 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 0 2026-03-20 16:26:25.670139 | controller | == 2 ']'\n logger.go:42: 16:23:22 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670144 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.670149 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-20 16:26:25.670153 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-20 16:26:25.670157 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-20 16:26:25.670161 | controller | 16:23:22 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.670167 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:23:22 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670172 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-20 16:26:25.670178 | controller | \ logger.go:42: 16:23:22 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-20 16:26:25.670184 | controller | error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found 2026-03-20 16:26:25.670189 | controller | in namespace \"watcher-kuttl-default\"\n logger.go:42: 16:23:22 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670195 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 16:23:23 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670200 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.670204 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-20 16:26:25.670208 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-20 16:26:25.670212 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-20 16:26:25.670216 | controller | 16:23:23 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.670220 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:23:23 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670225 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-20 16:26:25.670229 | controller | \ logger.go:42: 16:23:23 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-20 16:26:25.670233 | controller | error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found 2026-03-20 16:26:25.670239 | controller | in namespace \"watcher-kuttl-default\"\n logger.go:42: 16:23:23 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670245 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 16:23:24 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670250 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.670256 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-20 16:26:25.670262 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-20 16:26:25.670268 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-20 16:26:25.670273 | controller | 16:23:24 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.670279 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:23:24 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670284 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-20 16:26:25.670288 | controller | \ logger.go:42: 16:23:24 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-20 16:26:25.670292 | controller | error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found 2026-03-20 16:26:25.670296 | controller | in namespace \"watcher-kuttl-default\"\n logger.go:42: 16:23:24 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670304 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 16:23:25 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670310 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.670315 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-20 16:26:25.670321 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-20 16:26:25.670337 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-20 16:26:25.670342 | controller | 16:23:25 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.670346 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:23:25 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670351 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-20 16:26:25.670355 | controller | \ logger.go:42: 16:23:25 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-20 16:26:25.670362 | controller | error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found 2026-03-20 16:26:25.670366 | controller | in namespace \"watcher-kuttl-default\"\n logger.go:42: 16:23:25 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670372 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 16:23:26 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670378 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.670384 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-20 16:26:25.670390 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-20 16:26:25.670395 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-20 16:26:25.670399 | controller | 16:23:26 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.670403 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:23:26 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670407 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-20 16:26:25.670411 | controller | \ logger.go:42: 16:23:27 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-20 16:26:25.670416 | controller | error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found 2026-03-20 16:26:25.670420 | controller | in namespace \"watcher-kuttl-default\"\n logger.go:42: 16:23:27 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670424 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 16:23:28 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670428 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.670432 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-20 16:26:25.670436 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-20 16:26:25.670441 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-20 16:26:25.670445 | controller | 16:23:28 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c 'Block storage 2026-03-20 16:26:25.670449 | controller | service is not enabled, skipping storage collector'\n logger.go:42: 16:23:28 2026-03-20 16:26:25.670453 | controller | | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.670457 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:23:28 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670461 | controller | | error: error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" 2026-03-20 16:26:25.670466 | controller | not found in namespace \"watcher-kuttl-default\"\n logger.go:42: 16:23:28 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670470 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 16:23:29 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670477 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.670481 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-20 16:26:25.670485 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-20 16:26:25.670490 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-20 16:26:25.670494 | controller | 16:23:29 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.670498 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:23:29 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670502 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-20 16:26:25.670506 | controller | \ logger.go:42: 16:23:29 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-20 16:26:25.670511 | controller | error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found 2026-03-20 16:26:25.670516 | controller | in namespace \"watcher-kuttl-default\"\n logger.go:42: 16:23:29 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670521 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 16:23:30 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670527 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.670533 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-20 16:26:25.670538 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-20 16:26:25.670553 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-20 16:26:25.670560 | controller | 16:23:30 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.670565 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:23:30 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670569 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-20 16:26:25.670573 | controller | \ logger.go:42: 16:23:30 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-20 16:26:25.670577 | controller | error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found 2026-03-20 16:26:25.670581 | controller | in namespace \"watcher-kuttl-default\"\n logger.go:42: 16:23:30 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670586 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 16:23:31 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670590 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.670596 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-20 16:26:25.670600 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-20 16:26:25.670606 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-20 16:26:25.670611 | controller | 16:23:31 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.670617 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:23:31 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670622 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-20 16:26:25.670626 | controller | \ logger.go:42: 16:23:31 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-20 16:26:25.670641 | controller | error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found 2026-03-20 16:26:25.670645 | controller | in namespace \"watcher-kuttl-default\"\n logger.go:42: 16:23:31 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670650 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 16:23:32 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670656 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.670660 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-20 16:26:25.670665 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-20 16:26:25.670669 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-20 16:26:25.670673 | controller | 16:23:32 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.670677 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:23:32 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670681 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-20 16:26:25.670685 | controller | \ logger.go:42: 16:23:32 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-20 16:26:25.670690 | controller | error from server (NotFound): pods \"watcher-kuttl-decision-engine-0\" not found 2026-03-20 16:26:25.670711 | controller | in namespace \"watcher-kuttl-default\"\n logger.go:42: 16:23:32 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670726 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 16:23:33 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670731 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.670735 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-20 16:26:25.670739 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-20 16:26:25.670743 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-20 16:26:25.670747 | controller | 16:23:33 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.670751 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:23:33 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670756 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-20 16:26:25.670760 | controller | \ logger.go:42: 16:23:33 | watcher-cinder/1-deploy-watcher-no-cinder | Error from 2026-03-20 16:26:25.670764 | controller | server (BadRequest): container \"watcher-decision-engine\" in pod \"watcher-kuttl-decision-engine-0\" 2026-03-20 16:26:25.670769 | controller | is waiting to start: ContainerCreating\n logger.go:42: 16:23:33 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670775 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 16:23:34 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670780 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.670786 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-20 16:26:25.670799 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-20 16:26:25.670805 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-20 16:26:25.670820 | controller | 16:23:34 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c 'Block storage 2026-03-20 16:26:25.670827 | controller | service is not enabled, skipping storage collector'\n logger.go:42: 16:23:34 2026-03-20 16:26:25.670832 | controller | | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.670837 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:23:35 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670841 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 16:23:36 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670845 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.670849 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-20 16:26:25.670854 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-20 16:26:25.670860 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-20 16:26:25.670865 | controller | 16:23:36 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.670869 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:23:36 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670873 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-20 16:26:25.670879 | controller | \ logger.go:42: 16:23:36 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 0 2026-03-20 16:26:25.670884 | controller | == 2 ']'\n logger.go:42: 16:23:37 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670888 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.670892 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-20 16:26:25.670896 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-20 16:26:25.670900 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-20 16:26:25.670904 | controller | 16:23:37 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.670909 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:23:37 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670913 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-20 16:26:25.670917 | controller | \ logger.go:42: 16:23:37 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 1 2026-03-20 16:26:25.670921 | controller | == 2 ']'\n logger.go:42: 16:23:38 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670925 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.670929 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-20 16:26:25.670934 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-20 16:26:25.670938 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-20 16:26:25.670942 | controller | 16:23:38 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c 'Block storage 2026-03-20 16:26:25.670946 | controller | service is not enabled, skipping storage collector'\n logger.go:42: 16:23:38 2026-03-20 16:26:25.670950 | controller | | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.670954 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:23:38 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670959 | controller | | + '[' 2 == 2 ']'\n logger.go:42: 16:23:39 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.670965 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.670970 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-20 16:26:25.670976 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-20 16:26:25.670982 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-20 16:26:25.670987 | controller | 16:23:39 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep -c 'Block storage 2026-03-20 16:26:25.670992 | controller | service is not enabled, skipping storage collector'\n logger.go:42: 16:23:39 2026-03-20 16:26:25.670998 | controller | | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.671002 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:23:39 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.671006 | controller | | + '[' 2 == 2 ']'\n logger.go:42: 16:23:40 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.671010 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.671017 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-20 16:26:25.671021 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-20 16:26:25.671034 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-20 16:26:25.671039 | controller | 16:23:40 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.671043 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:23:40 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.671047 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-20 16:26:25.671051 | controller | \ logger.go:42: 16:23:41 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 2 2026-03-20 16:26:25.671056 | controller | == 2 ']'\n logger.go:42: 16:23:42 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.671060 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.671066 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-20 16:26:25.671070 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-20 16:26:25.671074 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-20 16:26:25.671079 | controller | 16:23:42 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.671083 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:23:42 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.671087 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-20 16:26:25.671091 | controller | \ logger.go:42: 16:23:42 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 2 2026-03-20 16:26:25.671095 | controller | == 2 ']'\n logger.go:42: 16:23:43 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.671099 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.671103 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-20 16:26:25.671108 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-20 16:26:25.671112 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-20 16:26:25.671116 | controller | 16:23:43 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.671126 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:23:43 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.671131 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-20 16:26:25.671135 | controller | \ logger.go:42: 16:23:43 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 2 2026-03-20 16:26:25.671139 | controller | == 2 ']'\n logger.go:42: 16:23:44 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.671144 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.671148 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-20 16:26:25.671152 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c 'Block storage service is 2026-03-20 16:26:25.671156 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-20 16:26:25.671160 | controller | 16:23:44 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.671165 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:23:44 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.671171 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-20 16:26:25.671176 | controller | \ logger.go:42: 16:23:44 | watcher-cinder/1-deploy-watcher-no-cinder | + '[' 2 2026-03-20 16:26:25.671185 | controller | == 2 ']'\n logger.go:42: 16:23:44 | watcher-cinder/1-deploy-watcher-no-cinder 2026-03-20 16:26:25.671190 | controller | | test step completed 1-deploy-watcher-no-cinder\n logger.go:42: 16:23:44 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.671196 | controller | | starting test step 2-deploy-cinder\n logger.go:42: 16:23:44 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.671200 | controller | | OpenStackControlPlane:watcher-kuttl-default/openstack updated\n logger.go:42: 2026-03-20 16:26:25.671205 | controller | 16:23:44 | watcher-cinder/2-deploy-cinder | running command: [sh -c set -euxo pipefail\n 2026-03-20 16:26:25.671209 | controller | \ # check that the decision detects that there is a cinder service and\n # 2026-03-20 16:26:25.671213 | controller | does not log that storage collector is skipped\n [ \"$(oc logs -n $NAMESPACE 2026-03-20 16:26:25.671227 | controller | watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is not enabled, 2026-03-20 16:26:25.671231 | controller | skipping storage collector')\" == 0 ]\n ]\n logger.go:42: 16:23:44 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.671235 | controller | | ++ oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.671240 | controller | 16:23:44 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.671245 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:23:45 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.671260 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:23:46 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.671267 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.671272 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.671277 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.671281 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.671289 | controller | 0 ]\n ]\n logger.go:42: 16:23:46 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.671293 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.671297 | controller | 16:23:46 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.671302 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:23:46 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.671306 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:23:47 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.671310 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.671314 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.671318 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.671324 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.671330 | controller | 0 ]\n ]\n logger.go:42: 16:23:47 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.671336 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.671341 | controller | 16:23:47 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.671346 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:23:47 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.671350 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:23:48 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.671354 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.671358 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.671362 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.671369 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.671373 | controller | 0 ]\n ]\n logger.go:42: 16:23:48 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.671377 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.671381 | controller | 16:23:48 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.671386 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:23:48 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.671390 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:23:49 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.671394 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.671398 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.671402 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.671406 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.671410 | controller | 0 ]\n ]\n logger.go:42: 16:23:49 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.671415 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.671419 | controller | 16:23:49 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.671423 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:23:49 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.671427 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:23:50 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.671432 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.671438 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.671443 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.671449 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.671453 | controller | 0 ]\n ]\n logger.go:42: 16:23:50 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.671457 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.671461 | controller | 16:23:50 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.671466 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:23:50 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.671481 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:23:51 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.671486 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.671490 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.671494 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.671499 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.671503 | controller | 0 ]\n ]\n logger.go:42: 16:23:51 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.671507 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.671511 | controller | 16:23:51 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.671515 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:23:52 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.671519 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:23:53 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.671523 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.671530 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.671534 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.671538 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.671542 | controller | 0 ]\n ]\n logger.go:42: 16:23:53 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.671546 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.671550 | controller | 16:23:53 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.671554 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:23:53 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.671559 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:23:54 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.671563 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.671588 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.671593 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.671598 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.671603 | controller | 0 ]\n ]\n logger.go:42: 16:23:54 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.671608 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.671613 | controller | 16:23:54 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.671618 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:23:54 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.671622 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:23:55 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.671627 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.671632 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.671637 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.671643 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.671648 | controller | 0 ]\n ]\n logger.go:42: 16:23:55 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.671654 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.671659 | controller | 16:23:55 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.671663 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:23:55 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.671667 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:23:56 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.671672 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.671676 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.671680 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.671687 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.671691 | controller | 0 ]\n ]\n logger.go:42: 16:23:56 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.671711 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.671715 | controller | 16:23:56 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.671719 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:23:57 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.671735 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:23:58 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.671740 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.671745 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.671751 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.671756 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.671762 | controller | 0 ]\n ]\n logger.go:42: 16:23:58 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.671767 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.671771 | controller | 16:23:58 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.671776 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:23:58 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.671780 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:23:59 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.671784 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.671788 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.671792 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.671797 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.671801 | controller | 0 ]\n ]\n logger.go:42: 16:23:59 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.671805 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.671810 | controller | 16:23:59 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.671818 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:23:59 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.671824 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:24:00 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.671830 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.671834 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.671838 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.671842 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.671846 | controller | 0 ]\n ]\n logger.go:42: 16:24:00 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.671850 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.671855 | controller | 16:24:00 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.671859 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:00 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.671863 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:24:01 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.671867 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.671872 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.671877 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.671883 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.671889 | controller | 0 ]\n ]\n logger.go:42: 16:24:01 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.671895 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.671903 | controller | 16:24:01 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.671909 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:01 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.671914 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:24:02 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.671920 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.671925 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.671931 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.671936 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.671942 | controller | 0 ]\n ]\n logger.go:42: 16:24:02 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.671946 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.671950 | controller | 16:24:02 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.671955 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:03 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.671959 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:24:04 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.671972 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.671977 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.671981 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.671985 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.671989 | controller | 0 ]\n ]\n logger.go:42: 16:24:04 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.671994 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.671998 | controller | 16:24:04 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.672002 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:04 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672007 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:24:05 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672012 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.672018 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.672023 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.672029 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.672033 | controller | 0 ]\n ]\n logger.go:42: 16:24:05 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.672037 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.672042 | controller | 16:24:05 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.672046 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:05 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672050 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:24:06 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672056 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.672061 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.672067 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.672071 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.672076 | controller | 0 ]\n ]\n logger.go:42: 16:24:06 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.672082 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.672086 | controller | 16:24:06 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.672091 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:06 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672095 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:24:07 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672099 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.672105 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.672110 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.672114 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.672118 | controller | 0 ]\n ]\n logger.go:42: 16:24:07 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.672122 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.672127 | controller | 16:24:07 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.672133 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:07 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672138 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:24:08 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672144 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.672148 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.672152 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.672158 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.672163 | controller | 0 ]\n ]\n logger.go:42: 16:24:08 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.672169 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.672175 | controller | 16:24:08 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.672180 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:09 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672186 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:24:10 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672201 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.672207 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.672212 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.672217 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.672222 | controller | 0 ]\n ]\n logger.go:42: 16:24:10 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.672228 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.672234 | controller | 16:24:10 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.672239 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:10 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672244 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:24:11 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672249 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.672253 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.672257 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.672264 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.672268 | controller | 0 ]\n ]\n logger.go:42: 16:24:11 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.672272 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.672276 | controller | 16:24:11 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.672280 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:11 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672284 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:24:12 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672288 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.672293 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.672297 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.672301 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.672305 | controller | 0 ]\n ]\n logger.go:42: 16:24:12 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.672309 | controller | grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 2026-03-20 16:26:25.672317 | controller | 16:24:12 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.672321 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:24:12 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672325 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:24:13 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672329 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.672343 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.672348 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.672352 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.672356 | controller | 0 ]\n ]\n logger.go:42: 16:24:13 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.672362 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.672368 | controller | 16:24:13 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.672373 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:13 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672379 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:24:14 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672384 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.672388 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.672392 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.672396 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.672400 | controller | 0 ]\n ]\n logger.go:42: 16:24:14 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.672404 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.672417 | controller | 16:24:14 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.672421 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:15 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672427 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:24:16 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672441 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.672446 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.672455 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.672459 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.672464 | controller | 0 ]\n ]\n logger.go:42: 16:24:16 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.672468 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.672472 | controller | 16:24:16 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.672488 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:16 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672492 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:24:17 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672496 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.672500 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.672504 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.672509 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.672513 | controller | 0 ]\n ]\n logger.go:42: 16:24:17 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.672517 | controller | grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 2026-03-20 16:26:25.672521 | controller | 16:24:17 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.672525 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:24:17 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672529 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:24:18 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672534 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.672538 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.672542 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.672550 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.672554 | controller | 0 ]\n ]\n logger.go:42: 16:24:18 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.672558 | controller | grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 2026-03-20 16:26:25.672562 | controller | 16:24:18 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.672566 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:24:18 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672571 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:24:19 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672575 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.672579 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.672583 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.672587 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.672591 | controller | 0 ]\n ]\n logger.go:42: 16:24:19 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.672595 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.672599 | controller | 16:24:19 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.672603 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:20 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672608 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:24:21 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672615 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.672619 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.672623 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.672627 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.672631 | controller | 0 ]\n ]\n logger.go:42: 16:24:21 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.672637 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.672642 | controller | 16:24:21 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.672646 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:21 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672650 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:24:22 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672662 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.672667 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.672671 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.672675 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.672680 | controller | 0 ]\n ]\n logger.go:42: 16:24:22 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.672684 | controller | grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 2026-03-20 16:26:25.672688 | controller | 16:24:22 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.672705 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:24:22 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672712 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:24:23 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672717 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.672721 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.672725 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.672729 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.672733 | controller | 0 ]\n ]\n logger.go:42: 16:24:23 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.672737 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.672741 | controller | 16:24:23 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.672745 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:23 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672750 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:24:24 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672756 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.672760 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.672764 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.672769 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.672773 | controller | 0 ]\n ]\n logger.go:42: 16:24:24 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.672777 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.672781 | controller | 16:24:24 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.672787 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:25 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672792 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:24:26 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672796 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.672800 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.672804 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.672808 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.672812 | controller | 0 ]\n ]\n logger.go:42: 16:24:26 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.672817 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.672821 | controller | 16:24:26 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.672825 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:26 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672829 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:24:27 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672833 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.672837 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.672842 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.672846 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.672850 | controller | 0 ]\n ]\n logger.go:42: 16:24:27 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.672854 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.672858 | controller | 16:24:27 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.672862 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:27 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672866 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:24:28 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672871 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.672884 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.672888 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.672892 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.672897 | controller | 0 ]\n ]\n logger.go:42: 16:24:28 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.672901 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.672905 | controller | 16:24:28 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.672909 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:28 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672914 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:24:29 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672918 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.672922 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.672926 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.672930 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.672937 | controller | 0 ]\n ]\n logger.go:42: 16:24:29 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.672943 | controller | grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 2026-03-20 16:26:25.672947 | controller | 16:24:29 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.672951 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:24:29 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672955 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:24:30 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672960 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.672964 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.672969 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.672974 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.672978 | controller | 0 ]\n ]\n logger.go:42: 16:24:30 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.672982 | controller | grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 2026-03-20 16:26:25.672986 | controller | 16:24:30 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.672990 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:24:31 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672995 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:24:32 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.672999 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.673003 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.673007 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.673011 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.673015 | controller | 0 ]\n ]\n logger.go:42: 16:24:32 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.673020 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.673024 | controller | 16:24:32 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.673028 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:32 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673032 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:24:33 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673036 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.673040 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.673045 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.673049 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.673053 | controller | 0 ]\n ]\n logger.go:42: 16:24:33 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.673057 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.673061 | controller | 16:24:33 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.673065 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:33 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673070 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:24:34 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673074 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.673085 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.673090 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.673094 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.673100 | controller | 0 ]\n ]\n logger.go:42: 16:24:34 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.673105 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.673118 | controller | 16:24:34 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.673122 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:34 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673126 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:24:35 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673131 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.673137 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.673141 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.673145 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.673150 | controller | 0 ]\n ]\n logger.go:42: 16:24:35 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.673154 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.673158 | controller | 16:24:35 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.673162 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:36 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673166 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:24:37 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673170 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.673175 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.673179 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.673183 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.673187 | controller | 0 ]\n ]\n logger.go:42: 16:24:37 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.673191 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.673195 | controller | 16:24:37 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.673200 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:37 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673204 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:24:38 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673208 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.673212 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.673216 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.673220 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.673225 | controller | 0 ]\n ]\n logger.go:42: 16:24:38 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.673229 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.673233 | controller | 16:24:38 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.673237 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:38 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673241 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:24:39 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673245 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.673250 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.673256 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.673260 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.673264 | controller | 0 ]\n ]\n logger.go:42: 16:24:39 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.673268 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.673272 | controller | 16:24:39 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.673276 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:39 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673281 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:24:40 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673285 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.673297 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.673301 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.673305 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.673310 | controller | 0 ]\n ]\n logger.go:42: 16:24:40 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.673314 | controller | grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 2026-03-20 16:26:25.673320 | controller | 16:24:40 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.673324 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:24:41 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673328 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:24:42 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673332 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.673337 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.673341 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.673345 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.673349 | controller | 0 ]\n ]\n logger.go:42: 16:24:42 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.673353 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.673357 | controller | 16:24:42 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.673361 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:42 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673366 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:24:43 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673370 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.673374 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.673378 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.673382 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.673386 | controller | 0 ]\n ]\n logger.go:42: 16:24:43 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.673391 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.673395 | controller | 16:24:43 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.673399 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:43 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673403 | controller | | + '[' 2 == 0 ']'\n logger.go:42: 16:24:44 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673409 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.673413 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.673417 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.673421 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.673426 | controller | 0 ]\n ]\n logger.go:42: 16:24:44 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.673430 | controller | grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 2026-03-20 16:26:25.673434 | controller | 16:24:44 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.673438 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:24:44 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673442 | controller | | Error from server (BadRequest): container \"watcher-decision-engine\" in pod \"watcher-kuttl-decision-engine-0\" 2026-03-20 16:26:25.673446 | controller | is waiting to start: ContainerCreating\n logger.go:42: 16:24:44 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673451 | controller | | + '[' 0 == 0 ']'\n logger.go:42: 16:24:45 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673455 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.673459 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.673465 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.673469 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.673473 | controller | 0 ]\n ]\n logger.go:42: 16:24:45 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.673477 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.673482 | controller | 16:24:45 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.673486 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:45 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673497 | controller | | Error from server (BadRequest): container \"watcher-decision-engine\" in pod \"watcher-kuttl-decision-engine-0\" 2026-03-20 16:26:25.673502 | controller | is waiting to start: ContainerCreating\n logger.go:42: 16:24:45 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673506 | controller | | + '[' 0 == 0 ']'\n logger.go:42: 16:24:46 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673510 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.673515 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.673519 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.673525 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.673529 | controller | 0 ]\n ]\n logger.go:42: 16:24:46 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.673533 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.673537 | controller | 16:24:46 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.673541 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:46 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673545 | controller | | + '[' 0 == 0 ']'\n logger.go:42: 16:24:47 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673550 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.673554 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.673558 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.673565 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.673569 | controller | 0 ]\n ]\n logger.go:42: 16:24:47 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.673574 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.673578 | controller | 16:24:47 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.673582 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:48 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673586 | controller | | + '[' 0 == 0 ']'\n logger.go:42: 16:24:49 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673590 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.673595 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.673599 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.673603 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.673607 | controller | 0 ]\n ]\n logger.go:42: 16:24:49 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.673611 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.673616 | controller | 16:24:49 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.673620 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:49 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673624 | controller | | + '[' 0 == 0 ']'\n logger.go:42: 16:24:50 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673628 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.673632 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.673637 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.673641 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.673645 | controller | 0 ]\n ]\n logger.go:42: 16:24:50 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.673649 | controller | grep -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 2026-03-20 16:26:25.673654 | controller | 16:24:50 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.673658 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:24:50 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673662 | controller | | + '[' 0 == 0 ']'\n logger.go:42: 16:24:51 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673666 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.673670 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.673674 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.673679 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.673683 | controller | 0 ]\n ]\n logger.go:42: 16:24:51 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.673687 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.673710 | controller | 16:24:51 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.673718 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:51 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673732 | controller | | + '[' 0 == 0 ']'\n logger.go:42: 16:24:52 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673738 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.673745 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.673749 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.673753 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.673758 | controller | 0 ]\n ]\n logger.go:42: 16:24:52 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.673762 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.673766 | controller | 16:24:52 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.673770 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:52 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673775 | controller | | + '[' 0 == 0 ']'\n logger.go:42: 16:24:53 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673779 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.673783 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.673796 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.673801 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.673805 | controller | 0 ]\n ]\n logger.go:42: 16:24:53 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.673809 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.673813 | controller | 16:24:53 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.673818 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:54 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673822 | controller | | + '[' 0 == 0 ']'\n logger.go:42: 16:24:55 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673826 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.673830 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.673835 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.673839 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.673843 | controller | 0 ]\n ]\n logger.go:42: 16:24:55 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.673847 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.673852 | controller | 16:24:55 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.673864 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:55 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673868 | controller | | + '[' 0 == 0 ']'\n logger.go:42: 16:24:56 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673872 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.673876 | controller | detects that there is a cinder service and\n # does not log that storage 2026-03-20 16:26:25.673881 | controller | collector is skipped\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.673885 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.673889 | controller | 0 ]\n ]\n logger.go:42: 16:24:56 | watcher-cinder/2-deploy-cinder | ++ 2026-03-20 16:26:25.673893 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.673898 | controller | 16:24:56 | watcher-cinder/2-deploy-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.673902 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:56 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673908 | controller | | + '[' 0 == 0 ']'\n logger.go:42: 16:24:56 | watcher-cinder/2-deploy-cinder 2026-03-20 16:26:25.673912 | controller | | test step completed 2-deploy-cinder\n logger.go:42: 16:24:56 | watcher-cinder/3-remove-cinder 2026-03-20 16:26:25.673916 | controller | | starting test step 3-remove-cinder\n logger.go:42: 16:24:56 | watcher-cinder/3-remove-cinder 2026-03-20 16:26:25.673928 | controller | | OpenStackControlPlane:watcher-kuttl-default/openstack updated\n logger.go:42: 2026-03-20 16:26:25.673933 | controller | 16:24:56 | watcher-cinder/3-remove-cinder | running command: [sh -c set -euxo pipefail\n 2026-03-20 16:26:25.673940 | controller | \ # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.673944 | controller | service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep 2026-03-20 16:26:25.673957 | controller | -c 'Block storage service is not enabled, skipping storage collector')\" == 2 ]\n 2026-03-20 16:26:25.673962 | controller | \ ]\n logger.go:42: 16:24:56 | watcher-cinder/3-remove-cinder | ++ grep 2026-03-20 16:26:25.673966 | controller | -c 'Block storage service is not enabled, skipping storage collector'\n logger.go:42: 2026-03-20 16:26:25.673970 | controller | 16:24:56 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.673974 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:24:56 | watcher-cinder/3-remove-cinder 2026-03-20 16:26:25.673979 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 16:24:57 | watcher-cinder/3-remove-cinder 2026-03-20 16:26:25.673983 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.673994 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-20 16:26:25.673999 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is 2026-03-20 16:26:25.674003 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-20 16:26:25.674007 | controller | 16:24:57 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.674011 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:24:57 | watcher-cinder/3-remove-cinder 2026-03-20 16:26:25.674016 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-20 16:26:25.674020 | controller | \ logger.go:42: 16:24:58 | watcher-cinder/3-remove-cinder | + '[' 0 == 2 ']'\n 2026-03-20 16:26:25.674024 | controller | \ logger.go:42: 16:24:59 | watcher-cinder/3-remove-cinder | running command: [sh 2026-03-20 16:26:25.674028 | controller | -c set -euxo pipefail\n # check that the decision engine correctly detects 2026-03-20 16:26:25.674033 | controller | that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.674037 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.674041 | controller | 2 ]\n ]\n logger.go:42: 16:24:59 | watcher-cinder/3-remove-cinder | ++ 2026-03-20 16:26:25.674045 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.674049 | controller | 16:24:59 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.674053 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:24:59 | watcher-cinder/3-remove-cinder 2026-03-20 16:26:25.674058 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 16:25:00 | watcher-cinder/3-remove-cinder 2026-03-20 16:26:25.674062 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.674066 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-20 16:26:25.674070 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is 2026-03-20 16:26:25.674075 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-20 16:26:25.674079 | controller | 16:25:00 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.674096 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:25:00 | watcher-cinder/3-remove-cinder 2026-03-20 16:26:25.674101 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-20 16:26:25.674105 | controller | \ logger.go:42: 16:25:00 | watcher-cinder/3-remove-cinder | + '[' 0 == 2 ']'\n 2026-03-20 16:26:25.674109 | controller | \ logger.go:42: 16:25:01 | watcher-cinder/3-remove-cinder | running command: [sh 2026-03-20 16:26:25.674114 | controller | -c set -euxo pipefail\n # check that the decision engine correctly detects 2026-03-20 16:26:25.674118 | controller | that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.674122 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.674129 | controller | 2 ]\n ]\n logger.go:42: 16:25:01 | watcher-cinder/3-remove-cinder | ++ 2026-03-20 16:26:25.674133 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.674138 | controller | 16:25:01 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.674142 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:25:01 | watcher-cinder/3-remove-cinder 2026-03-20 16:26:25.674146 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 16:25:02 | watcher-cinder/3-remove-cinder 2026-03-20 16:26:25.674150 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.674155 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-20 16:26:25.674159 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is 2026-03-20 16:26:25.674163 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-20 16:26:25.674167 | controller | 16:25:02 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.674179 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:25:02 | watcher-cinder/3-remove-cinder 2026-03-20 16:26:25.674184 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-20 16:26:25.674189 | controller | \ logger.go:42: 16:25:02 | watcher-cinder/3-remove-cinder | + '[' 0 == 2 ']'\n 2026-03-20 16:26:25.674193 | controller | \ logger.go:42: 16:25:03 | watcher-cinder/3-remove-cinder | running command: [sh 2026-03-20 16:26:25.674197 | controller | -c set -euxo pipefail\n # check that the decision engine correctly detects 2026-03-20 16:26:25.674202 | controller | that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.674206 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.674210 | controller | 2 ]\n ]\n logger.go:42: 16:25:03 | watcher-cinder/3-remove-cinder | ++ 2026-03-20 16:26:25.674214 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.674219 | controller | 16:25:03 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.674223 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:25:04 | watcher-cinder/3-remove-cinder 2026-03-20 16:26:25.674227 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 16:25:05 | watcher-cinder/3-remove-cinder 2026-03-20 16:26:25.674232 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.674236 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-20 16:26:25.674241 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is 2026-03-20 16:26:25.674245 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-20 16:26:25.674249 | controller | 16:25:05 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.674255 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:25:05 | watcher-cinder/3-remove-cinder 2026-03-20 16:26:25.674260 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-20 16:26:25.674264 | controller | \ logger.go:42: 16:25:05 | watcher-cinder/3-remove-cinder | + '[' 0 == 2 ']'\n 2026-03-20 16:26:25.674268 | controller | \ logger.go:42: 16:25:06 | watcher-cinder/3-remove-cinder | running command: [sh 2026-03-20 16:26:25.674272 | controller | -c set -euxo pipefail\n # check that the decision engine correctly detects 2026-03-20 16:26:25.674276 | controller | that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.674281 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.674285 | controller | 2 ]\n ]\n logger.go:42: 16:25:06 | watcher-cinder/3-remove-cinder | ++ 2026-03-20 16:26:25.674289 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.674295 | controller | 16:25:06 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.674300 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:25:06 | watcher-cinder/3-remove-cinder 2026-03-20 16:26:25.674305 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 16:25:07 | watcher-cinder/3-remove-cinder 2026-03-20 16:26:25.674310 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.674314 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-20 16:26:25.674318 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is 2026-03-20 16:26:25.674322 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-20 16:26:25.674326 | controller | 16:25:07 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.674331 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:25:07 | watcher-cinder/3-remove-cinder 2026-03-20 16:26:25.674335 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-20 16:26:25.674339 | controller | \ logger.go:42: 16:25:07 | watcher-cinder/3-remove-cinder | + '[' 0 == 2 ']'\n 2026-03-20 16:26:25.674343 | controller | \ logger.go:42: 16:25:08 | watcher-cinder/3-remove-cinder | running command: [sh 2026-03-20 16:26:25.674347 | controller | -c set -euxo pipefail\n # check that the decision engine correctly detects 2026-03-20 16:26:25.674352 | controller | that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.674356 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.674360 | controller | 2 ]\n ]\n logger.go:42: 16:25:08 | watcher-cinder/3-remove-cinder | ++ 2026-03-20 16:26:25.674364 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.674368 | controller | 16:25:08 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.674373 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:25:08 | watcher-cinder/3-remove-cinder 2026-03-20 16:26:25.674384 | controller | | Error from server (BadRequest): container \"watcher-decision-engine\" in pod \"watcher-kuttl-decision-engine-0\" 2026-03-20 16:26:25.674389 | controller | is waiting to start: ContainerCreating\n logger.go:42: 16:25:08 | watcher-cinder/3-remove-cinder 2026-03-20 16:26:25.674393 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 16:25:09 | watcher-cinder/3-remove-cinder 2026-03-20 16:26:25.674397 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.674402 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-20 16:26:25.674406 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is 2026-03-20 16:26:25.674412 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-20 16:26:25.674416 | controller | 16:25:09 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.674420 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:25:09 | watcher-cinder/3-remove-cinder 2026-03-20 16:26:25.674425 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-20 16:26:25.674429 | controller | \ logger.go:42: 16:25:09 | watcher-cinder/3-remove-cinder | Error from server 2026-03-20 16:26:25.674442 | controller | (BadRequest): container \"watcher-decision-engine\" in pod \"watcher-kuttl-decision-engine-0\" 2026-03-20 16:26:25.674447 | controller | is waiting to start: ContainerCreating\n logger.go:42: 16:25:09 | watcher-cinder/3-remove-cinder 2026-03-20 16:26:25.674451 | controller | | + '[' 0 == 2 ']'\n logger.go:42: 16:25:10 | watcher-cinder/3-remove-cinder 2026-03-20 16:26:25.674455 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.674459 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-20 16:26:25.674464 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is 2026-03-20 16:26:25.674468 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-20 16:26:25.674472 | controller | 16:25:10 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.674476 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:25:10 | watcher-cinder/3-remove-cinder 2026-03-20 16:26:25.674480 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-20 16:26:25.674485 | controller | \ logger.go:42: 16:25:11 | watcher-cinder/3-remove-cinder | + '[' 0 == 2 ']'\n 2026-03-20 16:26:25.674489 | controller | \ logger.go:42: 16:25:12 | watcher-cinder/3-remove-cinder | running command: [sh 2026-03-20 16:26:25.674495 | controller | -c set -euxo pipefail\n # check that the decision engine correctly detects 2026-03-20 16:26:25.674500 | controller | that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.674504 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.674508 | controller | 2 ]\n ]\n logger.go:42: 16:25:12 | watcher-cinder/3-remove-cinder | ++ 2026-03-20 16:26:25.674512 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.674517 | controller | 16:25:12 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.674522 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:25:12 | watcher-cinder/3-remove-cinder 2026-03-20 16:26:25.674526 | controller | | + '[' 1 == 2 ']'\n logger.go:42: 16:25:13 | watcher-cinder/3-remove-cinder 2026-03-20 16:26:25.674530 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.674534 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-20 16:26:25.674539 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is 2026-03-20 16:26:25.674543 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-20 16:26:25.674547 | controller | 16:25:13 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.674551 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:25:13 | watcher-cinder/3-remove-cinder 2026-03-20 16:26:25.674555 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-20 16:26:25.674560 | controller | \ logger.go:42: 16:25:13 | watcher-cinder/3-remove-cinder | + '[' 2 == 2 ']'\n 2026-03-20 16:26:25.674564 | controller | \ logger.go:42: 16:25:14 | watcher-cinder/3-remove-cinder | running command: [sh 2026-03-20 16:26:25.674570 | controller | -c set -euxo pipefail\n # check that the decision engine correctly detects 2026-03-20 16:26:25.674574 | controller | that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.674578 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.674583 | controller | 2 ]\n ]\n logger.go:42: 16:25:14 | watcher-cinder/3-remove-cinder | ++ 2026-03-20 16:26:25.674587 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.674598 | controller | 16:25:14 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.674603 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:25:14 | watcher-cinder/3-remove-cinder 2026-03-20 16:26:25.674607 | controller | | + '[' 2 == 2 ']'\n logger.go:42: 16:25:15 | watcher-cinder/3-remove-cinder 2026-03-20 16:26:25.674611 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.674616 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-20 16:26:25.674620 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is 2026-03-20 16:26:25.674624 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-20 16:26:25.674628 | controller | 16:25:15 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.674632 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:25:15 | watcher-cinder/3-remove-cinder 2026-03-20 16:26:25.674636 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-20 16:26:25.674642 | controller | \ logger.go:42: 16:25:15 | watcher-cinder/3-remove-cinder | + '[' 2 == 2 ']'\n 2026-03-20 16:26:25.674646 | controller | \ logger.go:42: 16:25:16 | watcher-cinder/3-remove-cinder | running command: [sh 2026-03-20 16:26:25.674650 | controller | -c set -euxo pipefail\n # check that the decision engine correctly detects 2026-03-20 16:26:25.674654 | controller | that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.674659 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.674663 | controller | 2 ]\n ]\n logger.go:42: 16:25:16 | watcher-cinder/3-remove-cinder | ++ 2026-03-20 16:26:25.674669 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.674673 | controller | 16:25:16 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.674678 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:25:17 | watcher-cinder/3-remove-cinder 2026-03-20 16:26:25.674682 | controller | | + '[' 2 == 2 ']'\n logger.go:42: 16:25:18 | watcher-cinder/3-remove-cinder 2026-03-20 16:26:25.674686 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.674690 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-20 16:26:25.674713 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is 2026-03-20 16:26:25.674717 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-20 16:26:25.674722 | controller | 16:25:18 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.674726 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:25:18 | watcher-cinder/3-remove-cinder 2026-03-20 16:26:25.674730 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-20 16:26:25.674734 | controller | \ logger.go:42: 16:25:18 | watcher-cinder/3-remove-cinder | + '[' 2 == 2 ']'\n 2026-03-20 16:26:25.674738 | controller | \ logger.go:42: 16:25:19 | watcher-cinder/3-remove-cinder | running command: [sh 2026-03-20 16:26:25.674746 | controller | -c set -euxo pipefail\n # check that the decision engine correctly detects 2026-03-20 16:26:25.674750 | controller | that there is no cinder service\n [ \"$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.674754 | controller | |grep -c 'Block storage service is not enabled, skipping storage collector')\" == 2026-03-20 16:26:25.674759 | controller | 2 ]\n ]\n logger.go:42: 16:25:19 | watcher-cinder/3-remove-cinder | ++ 2026-03-20 16:26:25.674763 | controller | oc logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0\n logger.go:42: 2026-03-20 16:26:25.674767 | controller | 16:25:19 | watcher-cinder/3-remove-cinder | ++ grep -c 'Block storage service is 2026-03-20 16:26:25.674771 | controller | not enabled, skipping storage collector'\n logger.go:42: 16:25:19 | watcher-cinder/3-remove-cinder 2026-03-20 16:26:25.674775 | controller | | + '[' 2 == 2 ']'\n logger.go:42: 16:25:20 | watcher-cinder/3-remove-cinder 2026-03-20 16:26:25.674780 | controller | | running command: [sh -c set -euxo pipefail\n # check that the decision 2026-03-20 16:26:25.674784 | controller | engine correctly detects that there is no cinder service\n [ \"$(oc logs 2026-03-20 16:26:25.674788 | controller | -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c 'Block storage service is 2026-03-20 16:26:25.674792 | controller | not enabled, skipping storage collector')\" == 2 ]\n ]\n logger.go:42: 2026-03-20 16:26:25.674796 | controller | 16:25:20 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.674803 | controller | watcher-kuttl-decision-engine-0\n logger.go:42: 16:25:20 | watcher-cinder/3-remove-cinder 2026-03-20 16:26:25.674807 | controller | | ++ grep -c 'Block storage service is not enabled, skipping storage collector'\n 2026-03-20 16:26:25.674819 | controller | \ logger.go:42: 16:25:20 | watcher-cinder/3-remove-cinder | + '[' 2 == 2 ']'\n 2026-03-20 16:26:25.674824 | controller | \ logger.go:42: 16:25:20 | watcher-cinder/3-remove-cinder | test step completed 2026-03-20 16:26:25.674836 | controller | 3-remove-cinder\n logger.go:42: 16:25:20 | watcher-cinder/4-cleanup-watcher | 2026-03-20 16:26:25.674840 | controller | starting test step 4-cleanup-watcher\n logger.go:42: 16:25:27 | watcher-cinder/4-cleanup-watcher 2026-03-20 16:26:25.674845 | controller | | test step completed 4-cleanup-watcher\n logger.go:42: 16:25:27 | watcher-cinder 2026-03-20 16:26:25.674849 | controller | | skipping kubernetes event logging\n=== CONT kuttl/harness/watcher-api-scaling\n 2026-03-20 16:26:25.674853 | controller | \ logger.go:42: 16:25:27 | watcher-api-scaling | Skipping creation of user-supplied 2026-03-20 16:26:25.674858 | controller | namespace: watcher-kuttl-default\n logger.go:42: 16:25:27 | watcher-api-scaling/0-cleanup-watcher 2026-03-20 16:26:25.674862 | controller | | starting test step 0-cleanup-watcher\n logger.go:42: 16:25:27 | watcher-api-scaling/0-cleanup-watcher 2026-03-20 16:26:25.674866 | controller | | test step completed 0-cleanup-watcher\n logger.go:42: 16:25:27 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.674870 | controller | | starting test step 1-deploy-with-defaults\n logger.go:42: 16:25:27 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.674874 | controller | | Watcher:watcher-kuttl-default/watcher-kuttl created\n logger.go:42: 16:25:28 2026-03-20 16:26:25.674879 | controller | | watcher-api-scaling/1-deploy-with-defaults | running command: [sh -c set -euxo 2026-03-20 16:26:25.674885 | controller | pipefail\n oc exec -n watcher-kuttl-default openstackclient -- openstack 2026-03-20 16:26:25.674889 | controller | service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n SERVICEID=$(oc 2026-03-20 16:26:25.674893 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-20 16:26:25.674898 | controller | -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n [ $(oc get -n watcher-kuttl-default 2026-03-20 16:26:25.674902 | controller | keystoneservice watcher -o jsonpath={.status.serviceID}) == ${SERVICEID} ]\n [ 2026-03-20 16:26:25.674906 | controller | -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})\" 2026-03-20 16:26:25.674912 | controller | ]\n # If we are running the container locally, skip following test\n if 2026-03-20 16:26:25.674916 | controller | [ \"$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)\" 2026-03-20 16:26:25.674921 | controller | == \"\" ]; then\n exit 0\n fi\n env_variables=$(oc set 2026-03-20 16:26:25.674925 | controller | env $(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher) 2026-03-20 16:26:25.674929 | controller | -n openstack-operators --list)\n counter=0\n for i in ${env_variables}; 2026-03-20 16:26:25.674933 | controller | do\n if echo ${i} | grep '_URL_DEFAULT' &> /dev/null; then\n echo 2026-03-20 16:26:25.674937 | controller | ${i}\n counter=$((counter + 1))\n fi\n done\n if 2026-03-20 16:26:25.674942 | controller | [ ${counter} -lt 3 ]; then\n echo \"Error: Less than 3 _URL_DEFAULT variables 2026-03-20 16:26:25.674946 | controller | found.\"\n exit 1\n else\n echo \"Success: ${counter} _URL_DEFAULT 2026-03-20 16:26:25.674950 | controller | variables found.\"\n fi\n ]\n logger.go:42: 16:25:28 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.674954 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-20 16:26:25.674961 | controller | value -c Name -c Type\n logger.go:42: 16:25:28 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.674966 | controller | | ++ grep -c '^watcher'\n logger.go:42: 16:25:30 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.674970 | controller | | + '[' 0 == 1 ']'\n logger.go:42: 16:25:31 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.674974 | controller | | running command: [sh -c set -euxo pipefail\n oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.674978 | controller | openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c 2026-03-20 16:26:25.674983 | controller | ^watcher) == 1 ]\n SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient 2026-03-20 16:26:25.674987 | controller | -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print 2026-03-20 16:26:25.674991 | controller | $1}')\n [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.674995 | controller | == ${SERVICEID} ]\n [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl 2026-03-20 16:26:25.675000 | controller | -o jsonpath={.status.hash.dbsync})\" ]\n # If we are running the container 2026-03-20 16:26:25.675004 | controller | locally, skip following test\n if [ \"$(oc get pods -n openstack-operators 2026-03-20 16:26:25.675008 | controller | -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 2026-03-20 16:26:25.675012 | controller | 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators 2026-03-20 16:26:25.675016 | controller | -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n 2026-03-20 16:26:25.675028 | controller | \ counter=0\n for i in ${env_variables}; do\n if echo ${i} 2026-03-20 16:26:25.675033 | controller | | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter 2026-03-20 16:26:25.675037 | controller | + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo 2026-03-20 16:26:25.675041 | controller | \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n 2026-03-20 16:26:25.675046 | controller | \ echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n 2026-03-20 16:26:25.675050 | controller | \ ]\n logger.go:42: 16:25:31 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675054 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-20 16:26:25.675059 | controller | value -c Name -c Type\n logger.go:42: 16:25:31 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675065 | controller | | ++ grep -c '^watcher'\n logger.go:42: 16:25:33 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675070 | controller | | + '[' 0 == 1 ']'\n logger.go:42: 16:25:34 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675074 | controller | | running command: [sh -c set -euxo pipefail\n oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.675078 | controller | openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c 2026-03-20 16:26:25.675083 | controller | ^watcher) == 1 ]\n SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient 2026-03-20 16:26:25.675087 | controller | -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print 2026-03-20 16:26:25.675091 | controller | $1}')\n [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.675095 | controller | == ${SERVICEID} ]\n [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl 2026-03-20 16:26:25.675099 | controller | -o jsonpath={.status.hash.dbsync})\" ]\n # If we are running the container 2026-03-20 16:26:25.675104 | controller | locally, skip following test\n if [ \"$(oc get pods -n openstack-operators 2026-03-20 16:26:25.675108 | controller | -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 2026-03-20 16:26:25.675114 | controller | 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators 2026-03-20 16:26:25.675118 | controller | -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n 2026-03-20 16:26:25.675123 | controller | \ counter=0\n for i in ${env_variables}; do\n if echo ${i} 2026-03-20 16:26:25.675127 | controller | | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter 2026-03-20 16:26:25.675131 | controller | + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo 2026-03-20 16:26:25.675135 | controller | \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n 2026-03-20 16:26:25.675139 | controller | \ echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n 2026-03-20 16:26:25.675144 | controller | \ ]\n logger.go:42: 16:25:34 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675148 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-20 16:26:25.675152 | controller | value -c Name -c Type\n logger.go:42: 16:25:34 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675156 | controller | | ++ grep -c '^watcher'\n logger.go:42: 16:25:36 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675161 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:25:36 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675165 | controller | | ++ grep watcher\n logger.go:42: 16:25:36 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675169 | controller | | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.675173 | controller | -f value -c Name -c Type -c ID\n logger.go:42: 16:25:36 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675177 | controller | | ++ awk '{print $1}'\n logger.go:42: 16:25:38 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675182 | controller | | + SERVICEID=fc1daa96ef8c455ca09bffb24ea64ff6\n logger.go:42: 16:25:38 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675186 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-20 16:26:25.675190 | controller | \ logger.go:42: 16:25:38 | watcher-api-scaling/1-deploy-with-defaults | + '[' 2026-03-20 16:26:25.675196 | controller | fc1daa96ef8c455ca09bffb24ea64ff6 == fc1daa96ef8c455ca09bffb24ea64ff6 ']'\n logger.go:42: 2026-03-20 16:26:25.675200 | controller | 16:25:38 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.675207 | controller | watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n logger.go:42: 16:25:38 2026-03-20 16:26:25.675211 | controller | | watcher-api-scaling/1-deploy-with-defaults | + '[' -n '' ']'\n logger.go:42: 2026-03-20 16:26:25.675216 | controller | 16:25:39 | watcher-api-scaling/1-deploy-with-defaults | running command: [sh -c 2026-03-20 16:26:25.675227 | controller | set -euxo pipefail\n oc exec -n watcher-kuttl-default openstackclient -- 2026-03-20 16:26:25.675232 | controller | openstack service list -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]\n 2026-03-20 16:26:25.675237 | controller | \ SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack 2026-03-20 16:26:25.675241 | controller | service list -f value -c Name -c Type -c ID | grep watcher| awk '{print $1}')\n 2026-03-20 16:26:25.675245 | controller | \ [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.675250 | controller | == ${SERVICEID} ]\n [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl 2026-03-20 16:26:25.675254 | controller | -o jsonpath={.status.hash.dbsync})\" ]\n # If we are running the container 2026-03-20 16:26:25.675258 | controller | locally, skip following test\n if [ \"$(oc get pods -n openstack-operators 2026-03-20 16:26:25.675262 | controller | -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 2026-03-20 16:26:25.675266 | controller | 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators 2026-03-20 16:26:25.675271 | controller | -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n 2026-03-20 16:26:25.675275 | controller | \ counter=0\n for i in ${env_variables}; do\n if echo ${i} 2026-03-20 16:26:25.675279 | controller | | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter 2026-03-20 16:26:25.675283 | controller | + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo 2026-03-20 16:26:25.675287 | controller | \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n 2026-03-20 16:26:25.675292 | controller | \ echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n 2026-03-20 16:26:25.675296 | controller | \ ]\n logger.go:42: 16:25:39 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675302 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-20 16:26:25.675307 | controller | value -c Name -c Type\n logger.go:42: 16:25:39 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675311 | controller | | ++ grep -c '^watcher'\n logger.go:42: 16:25:42 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675316 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:25:42 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675321 | controller | | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.675326 | controller | -f value -c Name -c Type -c ID\n logger.go:42: 16:25:42 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675330 | controller | | ++ awk '{print $1}'\n logger.go:42: 16:25:42 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675334 | controller | | ++ grep watcher\n logger.go:42: 16:25:44 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675339 | controller | | + SERVICEID=fc1daa96ef8c455ca09bffb24ea64ff6\n logger.go:42: 16:25:44 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675343 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-20 16:26:25.675347 | controller | \ logger.go:42: 16:25:44 | watcher-api-scaling/1-deploy-with-defaults | + '[' 2026-03-20 16:26:25.675351 | controller | fc1daa96ef8c455ca09bffb24ea64ff6 == fc1daa96ef8c455ca09bffb24ea64ff6 ']'\n logger.go:42: 2026-03-20 16:26:25.675356 | controller | 16:25:44 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.675360 | controller | watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n logger.go:42: 16:25:44 2026-03-20 16:26:25.675366 | controller | | watcher-api-scaling/1-deploy-with-defaults | + '[' -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.675373 | controller | ']'\n logger.go:42: 16:25:44 | watcher-api-scaling/1-deploy-with-defaults | ++ 2026-03-20 16:26:25.675377 | controller | oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n 2026-03-20 16:26:25.675382 | controller | \ logger.go:42: 16:25:45 | watcher-api-scaling/1-deploy-with-defaults | + '[' 2026-03-20 16:26:25.675386 | controller | '' == '' ']'\n logger.go:42: 16:25:45 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675390 | controller | | + exit 0\n logger.go:42: 16:25:46 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675394 | controller | | running command: [sh -c set -euxo pipefail\n oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.675399 | controller | openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c 2026-03-20 16:26:25.675403 | controller | ^watcher) == 1 ]\n SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient 2026-03-20 16:26:25.675407 | controller | -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print 2026-03-20 16:26:25.675411 | controller | $1}')\n [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.675416 | controller | == ${SERVICEID} ]\n [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl 2026-03-20 16:26:25.675426 | controller | -o jsonpath={.status.hash.dbsync})\" ]\n # If we are running the container 2026-03-20 16:26:25.675432 | controller | locally, skip following test\n if [ \"$(oc get pods -n openstack-operators 2026-03-20 16:26:25.675436 | controller | -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 2026-03-20 16:26:25.675440 | controller | 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators 2026-03-20 16:26:25.675445 | controller | -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n 2026-03-20 16:26:25.675449 | controller | \ counter=0\n for i in ${env_variables}; do\n if echo ${i} 2026-03-20 16:26:25.675453 | controller | | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter 2026-03-20 16:26:25.675457 | controller | + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo 2026-03-20 16:26:25.675462 | controller | \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n 2026-03-20 16:26:25.675468 | controller | \ echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n 2026-03-20 16:26:25.675473 | controller | \ ]\n logger.go:42: 16:25:46 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675479 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-20 16:26:25.675485 | controller | value -c Name -c Type\n logger.go:42: 16:25:46 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675491 | controller | | ++ grep -c '^watcher'\n logger.go:42: 16:25:47 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675497 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:25:47 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675503 | controller | | ++ grep watcher\n logger.go:42: 16:25:47 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675508 | controller | | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.675512 | controller | -f value -c Name -c Type -c ID\n logger.go:42: 16:25:47 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675517 | controller | | ++ awk '{print $1}'\n logger.go:42: 16:25:49 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675521 | controller | | + SERVICEID=fc1daa96ef8c455ca09bffb24ea64ff6\n logger.go:42: 16:25:49 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675528 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-20 16:26:25.675532 | controller | \ logger.go:42: 16:25:50 | watcher-api-scaling/1-deploy-with-defaults | + '[' 2026-03-20 16:26:25.675536 | controller | fc1daa96ef8c455ca09bffb24ea64ff6 == fc1daa96ef8c455ca09bffb24ea64ff6 ']'\n logger.go:42: 2026-03-20 16:26:25.675540 | controller | 16:25:50 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.675545 | controller | watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n logger.go:42: 16:25:50 2026-03-20 16:26:25.675549 | controller | | watcher-api-scaling/1-deploy-with-defaults | + '[' -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.675556 | controller | ']'\n logger.go:42: 16:25:50 | watcher-api-scaling/1-deploy-with-defaults | ++ 2026-03-20 16:26:25.675560 | controller | oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n 2026-03-20 16:26:25.675564 | controller | \ logger.go:42: 16:25:50 | watcher-api-scaling/1-deploy-with-defaults | + '[' 2026-03-20 16:26:25.675569 | controller | '' == '' ']'\n logger.go:42: 16:25:50 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675596 | controller | | + exit 0\n logger.go:42: 16:25:51 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675601 | controller | | running command: [sh -c set -euxo pipefail\n oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.675606 | controller | openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c 2026-03-20 16:26:25.675611 | controller | ^watcher) == 1 ]\n SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient 2026-03-20 16:26:25.675616 | controller | -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print 2026-03-20 16:26:25.675621 | controller | $1}')\n [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.675626 | controller | == ${SERVICEID} ]\n [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl 2026-03-20 16:26:25.675631 | controller | -o jsonpath={.status.hash.dbsync})\" ]\n # If we are running the container 2026-03-20 16:26:25.675636 | controller | locally, skip following test\n if [ \"$(oc get pods -n openstack-operators 2026-03-20 16:26:25.675641 | controller | -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 2026-03-20 16:26:25.675646 | controller | 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators 2026-03-20 16:26:25.675651 | controller | -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n 2026-03-20 16:26:25.675665 | controller | \ counter=0\n for i in ${env_variables}; do\n if echo ${i} 2026-03-20 16:26:25.675672 | controller | | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter 2026-03-20 16:26:25.675678 | controller | + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo 2026-03-20 16:26:25.675683 | controller | \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n 2026-03-20 16:26:25.675688 | controller | \ echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n 2026-03-20 16:26:25.675705 | controller | \ ]\n logger.go:42: 16:25:51 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675713 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-20 16:26:25.675718 | controller | value -c Name -c Type\n logger.go:42: 16:25:51 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675723 | controller | | ++ grep -c '^watcher'\n logger.go:42: 16:25:53 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675728 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:25:53 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675733 | controller | | ++ grep watcher\n logger.go:42: 16:25:53 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675740 | controller | | ++ awk '{print $1}'\n logger.go:42: 16:25:53 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675745 | controller | | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.675750 | controller | -f value -c Name -c Type -c ID\n logger.go:42: 16:25:55 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675755 | controller | | + SERVICEID=fc1daa96ef8c455ca09bffb24ea64ff6\n logger.go:42: 16:25:55 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675760 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-20 16:26:25.675765 | controller | \ logger.go:42: 16:25:55 | watcher-api-scaling/1-deploy-with-defaults | + '[' 2026-03-20 16:26:25.675770 | controller | fc1daa96ef8c455ca09bffb24ea64ff6 == fc1daa96ef8c455ca09bffb24ea64ff6 ']'\n logger.go:42: 2026-03-20 16:26:25.675775 | controller | 16:25:55 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.675780 | controller | watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n logger.go:42: 16:25:56 2026-03-20 16:26:25.675785 | controller | | watcher-api-scaling/1-deploy-with-defaults | + '[' -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.675794 | controller | ']'\n logger.go:42: 16:25:56 | watcher-api-scaling/1-deploy-with-defaults | ++ 2026-03-20 16:26:25.675799 | controller | oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n 2026-03-20 16:26:25.675804 | controller | \ logger.go:42: 16:25:56 | watcher-api-scaling/1-deploy-with-defaults | + '[' 2026-03-20 16:26:25.675809 | controller | '' == '' ']'\n logger.go:42: 16:25:56 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675813 | controller | | + exit 0\n logger.go:42: 16:25:57 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675818 | controller | | running command: [sh -c set -euxo pipefail\n oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.675824 | controller | openstackclient -- openstack service list -f value -c Name -c Type |[ $(grep -c 2026-03-20 16:26:25.675828 | controller | ^watcher) == 1 ]\n SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient 2026-03-20 16:26:25.675833 | controller | -- openstack service list -f value -c Name -c Type -c ID | grep watcher| awk '{print 2026-03-20 16:26:25.675837 | controller | $1}')\n [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.675841 | controller | == ${SERVICEID} ]\n [ -n \"$(oc get -n watcher-kuttl-default watcher watcher-kuttl 2026-03-20 16:26:25.675846 | controller | -o jsonpath={.status.hash.dbsync})\" ]\n # If we are running the container 2026-03-20 16:26:25.675850 | controller | locally, skip following test\n if [ \"$(oc get pods -n openstack-operators 2026-03-20 16:26:25.675854 | controller | -o name -l openstack.org/operator-name=watcher)\" == \"\" ]; then\n exit 2026-03-20 16:26:25.675858 | controller | 0\n fi\n env_variables=$(oc set env $(oc get pods -n openstack-operators 2026-03-20 16:26:25.675863 | controller | -o name -l openstack.org/operator-name=watcher) -n openstack-operators --list)\n 2026-03-20 16:26:25.675867 | controller | \ counter=0\n for i in ${env_variables}; do\n if echo ${i} 2026-03-20 16:26:25.675871 | controller | | grep '_URL_DEFAULT' &> /dev/null; then\n echo ${i}\n counter=$((counter 2026-03-20 16:26:25.675875 | controller | + 1))\n fi\n done\n if [ ${counter} -lt 3 ]; then\n echo 2026-03-20 16:26:25.675879 | controller | \"Error: Less than 3 _URL_DEFAULT variables found.\"\n exit 1\n else\n 2026-03-20 16:26:25.675884 | controller | \ echo \"Success: ${counter} _URL_DEFAULT variables found.\"\n fi\n 2026-03-20 16:26:25.675888 | controller | \ ]\n logger.go:42: 16:25:57 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675901 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-20 16:26:25.675906 | controller | value -c Name -c Type\n logger.go:42: 16:25:57 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675910 | controller | | ++ grep -c '^watcher'\n logger.go:42: 16:25:59 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675915 | controller | | + '[' 1 == 1 ']'\n logger.go:42: 16:25:59 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675919 | controller | | ++ oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.675923 | controller | -f value -c Name -c Type -c ID\n logger.go:42: 16:25:59 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675927 | controller | | ++ awk '{print $1}'\n logger.go:42: 16:25:59 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675932 | controller | | ++ grep watcher\n logger.go:42: 16:26:01 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675936 | controller | | + SERVICEID=fc1daa96ef8c455ca09bffb24ea64ff6\n logger.go:42: 16:26:01 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675940 | controller | | ++ oc get -n watcher-kuttl-default keystoneservice watcher -o 'jsonpath={.status.serviceID}'\n 2026-03-20 16:26:25.675944 | controller | \ logger.go:42: 16:26:01 | watcher-api-scaling/1-deploy-with-defaults | + '[' 2026-03-20 16:26:25.675949 | controller | fc1daa96ef8c455ca09bffb24ea64ff6 == fc1daa96ef8c455ca09bffb24ea64ff6 ']'\n logger.go:42: 2026-03-20 16:26:25.675953 | controller | 16:26:01 | watcher-api-scaling/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.675957 | controller | watcher watcher-kuttl -o 'jsonpath={.status.hash.dbsync}'\n logger.go:42: 16:26:01 2026-03-20 16:26:25.675961 | controller | | watcher-api-scaling/1-deploy-with-defaults | + '[' -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.675968 | controller | ']'\n logger.go:42: 16:26:01 | watcher-api-scaling/1-deploy-with-defaults | ++ 2026-03-20 16:26:25.675973 | controller | oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher\n 2026-03-20 16:26:25.675977 | controller | \ logger.go:42: 16:26:02 | watcher-api-scaling/1-deploy-with-defaults | + '[' 2026-03-20 16:26:25.675981 | controller | '' == '' ']'\n logger.go:42: 16:26:02 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675985 | controller | | + exit 0\n logger.go:42: 16:26:02 | watcher-api-scaling/1-deploy-with-defaults 2026-03-20 16:26:25.675990 | controller | | test step completed 1-deploy-with-defaults\n logger.go:42: 16:26:02 | watcher-api-scaling/2-scale-up-watcher-api 2026-03-20 16:26:25.675994 | controller | | starting test step 2-scale-up-watcher-api\n logger.go:42: 16:26:02 | watcher-api-scaling/2-scale-up-watcher-api 2026-03-20 16:26:25.675998 | controller | | running command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type='json' 2026-03-20 16:26:25.676002 | controller | -p='[{\"op\": \"replace\", \"path\": \"/spec/apiServiceTemplate/replicas\", \"value\":3}]'\n 2026-03-20 16:26:25.676006 | controller | \ ]\n logger.go:42: 16:26:02 | watcher-api-scaling/2-scale-up-watcher-api 2026-03-20 16:26:25.676011 | controller | | watcher.watcher.openstack.org/watcher-kuttl patched\n logger.go:42: 16:26:13 2026-03-20 16:26:25.676015 | controller | | watcher-api-scaling/2-scale-up-watcher-api | test step completed 2-scale-up-watcher-api\n 2026-03-20 16:26:25.676019 | controller | \ logger.go:42: 16:26:13 | watcher-api-scaling/3-scale-down-watcher-api | starting 2026-03-20 16:26:25.676023 | controller | test step 3-scale-down-watcher-api\n logger.go:42: 16:26:13 | watcher-api-scaling/3-scale-down-watcher-api 2026-03-20 16:26:25.676028 | controller | | running command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type='json' 2026-03-20 16:26:25.676032 | controller | -p='[{\"op\": \"replace\", \"path\": \"/spec/apiServiceTemplate/replicas\", \"value\":1}]'\n 2026-03-20 16:26:25.676036 | controller | \ ]\n logger.go:42: 16:26:13 | watcher-api-scaling/3-scale-down-watcher-api 2026-03-20 16:26:25.676043 | controller | | watcher.watcher.openstack.org/watcher-kuttl patched\n logger.go:42: 16:26:16 2026-03-20 16:26:25.676047 | controller | | watcher-api-scaling/3-scale-down-watcher-api | test step completed 3-scale-down-watcher-api\n 2026-03-20 16:26:25.676051 | controller | \ logger.go:42: 16:26:16 | watcher-api-scaling/4-scale-down-zero-watcher-api | 2026-03-20 16:26:25.676056 | controller | starting test step 4-scale-down-zero-watcher-api\n logger.go:42: 16:26:16 | watcher-api-scaling/4-scale-down-zero-watcher-api 2026-03-20 16:26:25.676060 | controller | | running command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type='json' 2026-03-20 16:26:25.676064 | controller | -p='[{\"op\": \"replace\", \"path\": \"/spec/apiServiceTemplate/replicas\", \"value\":0}]'\n 2026-03-20 16:26:25.676068 | controller | \ ]\n logger.go:42: 16:26:17 | watcher-api-scaling/4-scale-down-zero-watcher-api 2026-03-20 16:26:25.676073 | controller | | watcher.watcher.openstack.org/watcher-kuttl patched\n logger.go:42: 16:26:18 2026-03-20 16:26:25.676084 | controller | | watcher-api-scaling/4-scale-down-zero-watcher-api | test step completed 4-scale-down-zero-watcher-api\n 2026-03-20 16:26:25.676090 | controller | \ logger.go:42: 16:26:18 | watcher-api-scaling/5-cleanup-watcher | starting test 2026-03-20 16:26:25.676095 | controller | step 5-cleanup-watcher\n logger.go:42: 16:26:18 | watcher-api-scaling/5-cleanup-watcher 2026-03-20 16:26:25.676099 | controller | | running command: [sh -c set -ex\n oc exec -n watcher-kuttl-default openstackclient 2026-03-20 16:26:25.676104 | controller | -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 2026-03-20 16:26:25.676108 | controller | ]\n ]\n logger.go:42: 16:26:18 | watcher-api-scaling/5-cleanup-watcher 2026-03-20 16:26:25.676112 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-20 16:26:25.676116 | controller | value -c Name -c Type\n logger.go:42: 16:26:18 | watcher-api-scaling/5-cleanup-watcher 2026-03-20 16:26:25.676121 | controller | | ++ grep -c '^watcher'\n logger.go:42: 16:26:20 | watcher-api-scaling/5-cleanup-watcher 2026-03-20 16:26:25.676125 | controller | | + '[' 1 == 0 ']'\n logger.go:42: 16:26:22 | watcher-api-scaling/5-cleanup-watcher 2026-03-20 16:26:25.676129 | controller | | running command: [sh -c set -ex\n oc exec -n watcher-kuttl-default openstackclient 2026-03-20 16:26:25.676133 | controller | -- openstack service list -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 2026-03-20 16:26:25.676138 | controller | ]\n ]\n logger.go:42: 16:26:22 | watcher-api-scaling/5-cleanup-watcher 2026-03-20 16:26:25.676142 | controller | | + oc exec -n watcher-kuttl-default openstackclient -- openstack service list -f 2026-03-20 16:26:25.676146 | controller | value -c Name -c Type\n logger.go:42: 16:26:22 | watcher-api-scaling/5-cleanup-watcher 2026-03-20 16:26:25.676150 | controller | | ++ grep -c '^watcher'\n logger.go:42: 16:26:24 | watcher-api-scaling/5-cleanup-watcher 2026-03-20 16:26:25.676155 | controller | | + '[' 0 == 0 ']'\n logger.go:42: 16:26:24 | watcher-api-scaling/5-cleanup-watcher 2026-03-20 16:26:25.676159 | controller | | test step completed 5-cleanup-watcher\n logger.go:42: 16:26:24 | watcher-api-scaling 2026-03-20 16:26:25.676163 | controller | | skipping kubernetes event logging\n=== CONT kuttl/harness/deps\n logger.go:42: 2026-03-20 16:26:25.676167 | controller | 16:26:24 | deps | Ignoring infra.yaml as it does not match file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n 2026-03-20 16:26:25.676172 | controller | \ logger.go:42: 16:26:24 | deps | Ignoring keystone.yaml as it does not match 2026-03-20 16:26:25.676176 | controller | file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n logger.go:42: 16:26:24 | 2026-03-20 16:26:25.676180 | controller | deps | Ignoring kustomization.yaml as it does not match file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n 2026-03-20 16:26:25.676184 | controller | \ logger.go:42: 16:26:24 | deps | Ignoring namespace.yaml as it does not match 2026-03-20 16:26:25.676189 | controller | file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n logger.go:42: 16:26:24 | 2026-03-20 16:26:25.676196 | controller | deps | Ignoring telemetry.yaml as it does not match file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n 2026-03-20 16:26:25.676202 | controller | \ logger.go:42: 16:26:24 | deps | Skipping creation of user-supplied namespace: 2026-03-20 16:26:25.676206 | controller | watcher-kuttl-default\n logger.go:42: 16:26:24 | deps | skipping kubernetes event 2026-03-20 16:26:25.676211 | controller | logging\n=== CONT kuttl/harness/common\n logger.go:42: 16:26:24 | common | Ignoring 2026-03-20 16:26:25.676215 | controller | cleanup-assert.yaml as it does not match file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n 2026-03-20 16:26:25.676219 | controller | \ logger.go:42: 16:26:24 | common | Ignoring cleanup-errors.yaml as it does not 2026-03-20 16:26:25.676223 | controller | match file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n logger.go:42: 16:26:24 2026-03-20 16:26:25.676228 | controller | | common | Ignoring cleanup-watcher.yaml as it does not match file name regexp: 2026-03-20 16:26:25.676232 | controller | ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n logger.go:42: 16:26:24 | common | Ignoring 2026-03-20 16:26:25.676236 | controller | deploy-with-defaults.yaml as it does not match file name regexp: ^(\\d+)-(?:[^\\.]+)(?:\\.yaml)?$\n 2026-03-20 16:26:25.676240 | controller | \ logger.go:42: 16:26:24 | common | Skipping creation of user-supplied namespace: 2026-03-20 16:26:25.676244 | controller | watcher-kuttl-default\n logger.go:42: 16:26:24 | common | skipping kubernetes 2026-03-20 16:26:25.676249 | controller | event logging\n=== NAME kuttl\n harness.go:406: run tests finished\n harness.go:514: 2026-03-20 16:26:25.676253 | controller | cleaning up\n harness.go:571: removing temp folder: \"\"\n--- FAIL: kuttl (1493.95s)\n 2026-03-20 16:26:25.676257 | controller | \ --- FAIL: kuttl/harness (0.00s)\n --- FAIL: kuttl/harness/appcred-tests 2026-03-20 16:26:25.676261 | controller | (962.71s)\n --- PASS: kuttl/harness/watcher-notification (37.61s)\n --- 2026-03-20 16:26:25.676266 | controller | PASS: kuttl/harness/watcher-topology (31.57s)\n --- PASS: kuttl/harness/watcher-tls-certs-change 2026-03-20 16:26:25.676270 | controller | (36.81s)\n --- PASS: kuttl/harness/watcher-tls (127.41s)\n --- PASS: 2026-03-20 16:26:25.676274 | controller | kuttl/harness/watcher-rmquser (34.91s)\n --- PASS: kuttl/harness/watcher 2026-03-20 16:26:25.676286 | controller | (79.13s)\n --- PASS: kuttl/harness/watcher-cinder (126.78s)\n --- 2026-03-20 16:26:25.676290 | controller | PASS: kuttl/harness/watcher-api-scaling (56.98s)\n --- PASS: kuttl/harness/deps 2026-03-20 16:26:25.676295 | controller | (0.00s)\n --- PASS: kuttl/harness/common (0.00s)\nFAIL" 2026-03-20 16:26:25.676300 | controller | stdout_lines: 2026-03-20 16:26:25.676305 | controller | - oc kuttl test --v 1 --start-kind=false --config test/kuttl/test-suites/default/config.yaml 2026-03-20 16:26:25.676309 | controller | - === RUN kuttl 2026-03-20 16:26:25.676313 | controller | - ' harness.go:463: starting setup' 2026-03-20 16:26:25.676318 | controller | - ' harness.go:255: running tests using configured kubeconfig.' 2026-03-20 16:26:25.676322 | controller | - ' harness.go:278: Successful connection to cluster at: https://api.crc.testing:6443' 2026-03-20 16:26:25.676326 | controller | - ' harness.go:363: running tests' 2026-03-20 16:26:25.676331 | controller | - ' harness.go:75: going to run test suite with timeout of 300 seconds for each 2026-03-20 16:26:25.676335 | controller | step' 2026-03-20 16:26:25.676339 | controller | - ' harness.go:375: testsuite: test/kuttl/test-suites/default/ has 11 tests' 2026-03-20 16:26:25.676344 | controller | - === RUN kuttl/harness 2026-03-20 16:26:25.676348 | controller | - === RUN kuttl/harness/appcred-tests 2026-03-20 16:26:25.676352 | controller | - === PAUSE kuttl/harness/appcred-tests 2026-03-20 16:26:25.676356 | controller | - === RUN kuttl/harness/common 2026-03-20 16:26:25.676361 | controller | - === PAUSE kuttl/harness/common 2026-03-20 16:26:25.676365 | controller | - === RUN kuttl/harness/deps 2026-03-20 16:26:25.676369 | controller | - === PAUSE kuttl/harness/deps 2026-03-20 16:26:25.676373 | controller | - === RUN kuttl/harness/watcher 2026-03-20 16:26:25.676379 | controller | - === PAUSE kuttl/harness/watcher 2026-03-20 16:26:25.676384 | controller | - === RUN kuttl/harness/watcher-api-scaling 2026-03-20 16:26:25.676388 | controller | - === PAUSE kuttl/harness/watcher-api-scaling 2026-03-20 16:26:25.676392 | controller | - === RUN kuttl/harness/watcher-cinder 2026-03-20 16:26:25.676397 | controller | - === PAUSE kuttl/harness/watcher-cinder 2026-03-20 16:26:25.676401 | controller | - === RUN kuttl/harness/watcher-notification 2026-03-20 16:26:25.676405 | controller | - === PAUSE kuttl/harness/watcher-notification 2026-03-20 16:26:25.676409 | controller | - === RUN kuttl/harness/watcher-rmquser 2026-03-20 16:26:25.676414 | controller | - === PAUSE kuttl/harness/watcher-rmquser 2026-03-20 16:26:25.676418 | controller | - === RUN kuttl/harness/watcher-tls 2026-03-20 16:26:25.676425 | controller | - === PAUSE kuttl/harness/watcher-tls 2026-03-20 16:26:25.676429 | controller | - === RUN kuttl/harness/watcher-tls-certs-change 2026-03-20 16:26:25.676434 | controller | - === PAUSE kuttl/harness/watcher-tls-certs-change 2026-03-20 16:26:25.676438 | controller | - === RUN kuttl/harness/watcher-topology 2026-03-20 16:26:25.676442 | controller | - === PAUSE kuttl/harness/watcher-topology 2026-03-20 16:26:25.676446 | controller | - === CONT kuttl/harness/appcred-tests 2026-03-20 16:26:25.676451 | controller | - ' logger.go:42: 16:01:30 | appcred-tests | Skipping creation of user-supplied 2026-03-20 16:26:25.676455 | controller | namespace: watcher-kuttl-default' 2026-03-20 16:26:25.676459 | controller | - ' logger.go:42: 16:01:30 | appcred-tests/0-cleanup | starting test step 0-cleanup' 2026-03-20 16:26:25.676463 | controller | - ' logger.go:42: 16:01:30 | appcred-tests/0-cleanup | test step completed 0-cleanup' 2026-03-20 16:26:25.676468 | controller | - ' logger.go:42: 16:01:30 | appcred-tests/1-deploy | starting test step 1-deploy' 2026-03-20 16:26:25.676472 | controller | - ' logger.go:42: 16:01:31 | appcred-tests/1-deploy | Watcher:watcher-kuttl-default/watcher-kuttl 2026-03-20 16:26:25.676477 | controller | created' 2026-03-20 16:26:25.676482 | controller | - ' logger.go:42: 16:01:31 | appcred-tests/1-deploy | running command: [sh -c set 2026-03-20 16:26:25.676486 | controller | -euo pipefail' 2026-03-20 16:26:25.676490 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.676494 | controller | - ' oc wait -n "${NS}" watcher watcher-kuttl --for=condition=Ready --timeout=600s]' 2026-03-20 16:26:25.676499 | controller | - ' logger.go:42: 16:02:31 | appcred-tests/1-deploy | watcher.watcher.openstack.org/watcher-kuttl 2026-03-20 16:26:25.676503 | controller | condition met' 2026-03-20 16:26:25.676507 | controller | - ' logger.go:42: 16:02:31 | appcred-tests/1-deploy | test step completed 1-deploy' 2026-03-20 16:26:25.676511 | controller | - ' logger.go:42: 16:02:31 | appcred-tests/2-deploy-appcred | starting test step 2026-03-20 16:26:25.676515 | controller | 2-deploy-appcred' 2026-03-20 16:26:25.676520 | controller | - ' logger.go:42: 16:02:31 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.676524 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.676528 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.676532 | controller | - ' ' 2026-03-20 16:26:25.676537 | controller | - ' old_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.676541 | controller | - ' old_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.676545 | controller | - ' old_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.676549 | controller | - ' oc create configmap appcred-watcher-pre \' 2026-03-20 16:26:25.676553 | controller | - ' --from-literal=api_uid="${old_api_uid}" \' 2026-03-20 16:26:25.676558 | controller | - ' --from-literal=applier_uid="${old_applier_uid}" \' 2026-03-20 16:26:25.676562 | controller | - ' --from-literal=decision_engine_uid="${old_de_uid}" \' 2026-03-20 16:26:25.676566 | controller | - ' --dry-run=client -o yaml | oc apply -n "${NS}" -f -' 2026-03-20 16:26:25.676570 | controller | - ' ' 2026-03-20 16:26:25.676576 | controller | - ' cat <<''EOF'' | oc apply -n "${NS}" -f -' 2026-03-20 16:26:25.676582 | controller | - ' apiVersion: keystone.openstack.org/v1beta1' 2026-03-20 16:26:25.676586 | controller | - ' kind: KeystoneApplicationCredential' 2026-03-20 16:26:25.676590 | controller | - ' metadata:' 2026-03-20 16:26:25.676595 | controller | - ' name: ac-watcher' 2026-03-20 16:26:25.676599 | controller | - ' spec:' 2026-03-20 16:26:25.676603 | controller | - ' secret: osp-secret' 2026-03-20 16:26:25.676607 | controller | - ' passwordSelector: WatcherPassword' 2026-03-20 16:26:25.676619 | controller | - ' userName: watcher' 2026-03-20 16:26:25.676625 | controller | - ' roles:' 2026-03-20 16:26:25.676629 | controller | - ' - admin' 2026-03-20 16:26:25.676633 | controller | - ' - service' 2026-03-20 16:26:25.676638 | controller | - ' unrestricted: false' 2026-03-20 16:26:25.676642 | controller | - ' EOF' 2026-03-20 16:26:25.676646 | controller | - ' ' 2026-03-20 16:26:25.676650 | controller | - ' oc patch watcher watcher-kuttl -n "${NS}" --type=merge -p ''{"spec":{"auth":{"applicationCredentialSecret":"ac-watcher-secret"}}}'']' 2026-03-20 16:26:25.676654 | controller | - ' logger.go:42: 16:02:32 | appcred-tests/2-deploy-appcred | configmap/appcred-watcher-pre 2026-03-20 16:26:25.676658 | controller | created' 2026-03-20 16:26:25.676663 | controller | - ' logger.go:42: 16:02:32 | appcred-tests/2-deploy-appcred | keystoneapplicationcredential.keystone.openstack.org/ac-watcher 2026-03-20 16:26:25.676669 | controller | created' 2026-03-20 16:26:25.676673 | controller | - ' logger.go:42: 16:02:32 | appcred-tests/2-deploy-appcred | watcher.watcher.openstack.org/watcher-kuttl 2026-03-20 16:26:25.676677 | controller | patched' 2026-03-20 16:26:25.676682 | controller | - ' logger.go:42: 16:02:32 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.676686 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.676690 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.676708 | controller | - ' ' 2026-03-20 16:26:25.676712 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.676716 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.676721 | controller | - ' ' 2026-03-20 16:26:25.676725 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.676729 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.676733 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.676738 | controller | - ' exit 1' 2026-03-20 16:26:25.676742 | controller | - ' fi' 2026-03-20 16:26:25.676746 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.676750 | controller | - ' ' 2026-03-20 16:26:25.676755 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.676759 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.676763 | controller | | base64 -d)' 2026-03-20 16:26:25.676769 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.676774 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.676778 | controller | - ' exit 1' 2026-03-20 16:26:25.676782 | controller | - ' fi' 2026-03-20 16:26:25.676787 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.676791 | controller | - ' ' 2026-03-20 16:26:25.676795 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.676799 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.676804 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.676808 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.676812 | controller | --timeout=300s' 2026-03-20 16:26:25.676816 | controller | - ' ' 2026-03-20 16:26:25.676821 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.676827 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.676831 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.676835 | controller | - ' ' 2026-03-20 16:26:25.676840 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.676844 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.676848 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.676852 | controller | - ' ' 2026-03-20 16:26:25.676856 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.676861 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.676865 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.676869 | controller | available"' 2026-03-20 16:26:25.676874 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.676878 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.676882 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.676887 | controller | - ' exit 1' 2026-03-20 16:26:25.676891 | controller | - ' fi' 2026-03-20 16:26:25.676895 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.676899 | controller | - ' ' 2026-03-20 16:26:25.676904 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.676908 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.676912 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.676917 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.676929 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.676934 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.676941 | controller | -- \' 2026-03-20 16:26:25.676946 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.676950 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.676954 | controller | - ' logger.go:42: 16:02:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.676959 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.676963 | controller | - ' logger.go:42: 16:07:32 | appcred-tests/2-deploy-appcred | error: timed out 2026-03-20 16:26:25.676967 | controller | waiting for the condition on keystoneapplicationcredentials/ac-watcher' 2026-03-20 16:26:25.676972 | controller | - ' logger.go:42: 16:07:33 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.676976 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.676980 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.676984 | controller | - ' ' 2026-03-20 16:26:25.676989 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.676993 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.676997 | controller | - ' ' 2026-03-20 16:26:25.677001 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.677005 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.677011 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.677016 | controller | - ' exit 1' 2026-03-20 16:26:25.677020 | controller | - ' fi' 2026-03-20 16:26:25.677024 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.677029 | controller | - ' ' 2026-03-20 16:26:25.677033 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.677037 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.677041 | controller | | base64 -d)' 2026-03-20 16:26:25.677046 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.677050 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.677054 | controller | - ' exit 1' 2026-03-20 16:26:25.677059 | controller | - ' fi' 2026-03-20 16:26:25.677063 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.677069 | controller | - ' ' 2026-03-20 16:26:25.677073 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.677077 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.677082 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.677086 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.677090 | controller | --timeout=300s' 2026-03-20 16:26:25.677094 | controller | - ' ' 2026-03-20 16:26:25.677099 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.677103 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.677107 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.677111 | controller | - ' ' 2026-03-20 16:26:25.677115 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.677120 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.677124 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.677128 | controller | - ' ' 2026-03-20 16:26:25.677133 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.677137 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.677141 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.677145 | controller | available"' 2026-03-20 16:26:25.677150 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.677154 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.677158 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.677162 | controller | - ' exit 1' 2026-03-20 16:26:25.677167 | controller | - ' fi' 2026-03-20 16:26:25.677171 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.677175 | controller | - ' ' 2026-03-20 16:26:25.677179 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.677184 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.677188 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.677192 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.677196 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.677212 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.677217 | controller | -- \' 2026-03-20 16:26:25.677221 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.677225 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.677229 | controller | - ' logger.go:42: 16:07:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.677234 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.677238 | controller | - ' logger.go:42: 16:07:33 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.677242 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.677247 | controller | found' 2026-03-20 16:26:25.677251 | controller | - ' logger.go:42: 16:07:34 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.677255 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.677259 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.677263 | controller | - ' ' 2026-03-20 16:26:25.677268 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.677272 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.677276 | controller | - ' ' 2026-03-20 16:26:25.677280 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.677285 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.677289 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.677293 | controller | - ' exit 1' 2026-03-20 16:26:25.677297 | controller | - ' fi' 2026-03-20 16:26:25.677302 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.677306 | controller | - ' ' 2026-03-20 16:26:25.677310 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.677314 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.677319 | controller | | base64 -d)' 2026-03-20 16:26:25.677324 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.677329 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.677333 | controller | - ' exit 1' 2026-03-20 16:26:25.677337 | controller | - ' fi' 2026-03-20 16:26:25.677341 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.677346 | controller | - ' ' 2026-03-20 16:26:25.677350 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.677354 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.677358 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.677364 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.677369 | controller | --timeout=300s' 2026-03-20 16:26:25.677373 | controller | - ' ' 2026-03-20 16:26:25.677377 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.677381 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.677386 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.677390 | controller | - ' ' 2026-03-20 16:26:25.677394 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.677398 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.677405 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.677409 | controller | - ' ' 2026-03-20 16:26:25.677414 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.677418 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.677422 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.677426 | controller | available"' 2026-03-20 16:26:25.677430 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.677435 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.677439 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.677443 | controller | - ' exit 1' 2026-03-20 16:26:25.677448 | controller | - ' fi' 2026-03-20 16:26:25.677452 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.677456 | controller | - ' ' 2026-03-20 16:26:25.677460 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.677464 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.677469 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.677473 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.677477 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.677481 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.677486 | controller | -- \' 2026-03-20 16:26:25.677497 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.677502 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.677506 | controller | - ' logger.go:42: 16:07:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.677511 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.677515 | controller | - ' logger.go:42: 16:07:34 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.677519 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.677523 | controller | found' 2026-03-20 16:26:25.677528 | controller | - ' logger.go:42: 16:07:35 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.677532 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.677536 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.677540 | controller | - ' ' 2026-03-20 16:26:25.677545 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.677549 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.677553 | controller | - ' ' 2026-03-20 16:26:25.677558 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.677564 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.677570 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.677576 | controller | - ' exit 1' 2026-03-20 16:26:25.677583 | controller | - ' fi' 2026-03-20 16:26:25.677588 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.677593 | controller | - ' ' 2026-03-20 16:26:25.677599 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.677603 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.677608 | controller | | base64 -d)' 2026-03-20 16:26:25.677614 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.677619 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.677623 | controller | - ' exit 1' 2026-03-20 16:26:25.677628 | controller | - ' fi' 2026-03-20 16:26:25.677632 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.677636 | controller | - ' ' 2026-03-20 16:26:25.677641 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.677645 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.677649 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.677653 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.677658 | controller | --timeout=300s' 2026-03-20 16:26:25.677662 | controller | - ' ' 2026-03-20 16:26:25.677666 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.677670 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.677675 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.677679 | controller | - ' ' 2026-03-20 16:26:25.677683 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.677687 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.677705 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.677713 | controller | - ' ' 2026-03-20 16:26:25.677717 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.677722 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.677726 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.677730 | controller | available"' 2026-03-20 16:26:25.677734 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.677739 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.677743 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.677747 | controller | - ' exit 1' 2026-03-20 16:26:25.677751 | controller | - ' fi' 2026-03-20 16:26:25.677756 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.677760 | controller | - ' ' 2026-03-20 16:26:25.677764 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.677768 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.677773 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.677777 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.677782 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.677788 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.677794 | controller | -- \' 2026-03-20 16:26:25.677810 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.677817 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.677823 | controller | - ' logger.go:42: 16:07:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.677829 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.677834 | controller | - ' logger.go:42: 16:07:36 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.677838 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.677842 | controller | found' 2026-03-20 16:26:25.677847 | controller | - ' logger.go:42: 16:07:37 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.677851 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.677855 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.677859 | controller | - ' ' 2026-03-20 16:26:25.677864 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.677868 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.677872 | controller | - ' ' 2026-03-20 16:26:25.677876 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.677891 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.677895 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.677900 | controller | - ' exit 1' 2026-03-20 16:26:25.677904 | controller | - ' fi' 2026-03-20 16:26:25.677908 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.677912 | controller | - ' ' 2026-03-20 16:26:25.677917 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.677921 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.677925 | controller | | base64 -d)' 2026-03-20 16:26:25.677929 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.677933 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.677938 | controller | - ' exit 1' 2026-03-20 16:26:25.677942 | controller | - ' fi' 2026-03-20 16:26:25.677946 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.677950 | controller | - ' ' 2026-03-20 16:26:25.677954 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.677966 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.677970 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.677975 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.677979 | controller | --timeout=300s' 2026-03-20 16:26:25.677983 | controller | - ' ' 2026-03-20 16:26:25.677987 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.677991 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.677995 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.678000 | controller | - ' ' 2026-03-20 16:26:25.678004 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.678008 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.678015 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.678019 | controller | - ' ' 2026-03-20 16:26:25.678032 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.678037 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.678041 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.678045 | controller | available"' 2026-03-20 16:26:25.678052 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.678057 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.678061 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.678065 | controller | - ' exit 1' 2026-03-20 16:26:25.678069 | controller | - ' fi' 2026-03-20 16:26:25.678073 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.678078 | controller | - ' ' 2026-03-20 16:26:25.678082 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.678086 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.678090 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.678094 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.678098 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.678103 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.678107 | controller | -- \' 2026-03-20 16:26:25.678119 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.678133 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.678139 | controller | - ' logger.go:42: 16:07:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.678145 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.678151 | controller | - ' logger.go:42: 16:07:37 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.678157 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.678162 | controller | found' 2026-03-20 16:26:25.678166 | controller | - ' logger.go:42: 16:07:38 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.678170 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.678174 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.678179 | controller | - ' ' 2026-03-20 16:26:25.678183 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.678187 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.678191 | controller | - ' ' 2026-03-20 16:26:25.678195 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.678200 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.678204 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.678208 | controller | - ' exit 1' 2026-03-20 16:26:25.678212 | controller | - ' fi' 2026-03-20 16:26:25.678216 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.678221 | controller | - ' ' 2026-03-20 16:26:25.678225 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.678229 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.678233 | controller | | base64 -d)' 2026-03-20 16:26:25.678237 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.678242 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.678246 | controller | - ' exit 1' 2026-03-20 16:26:25.678250 | controller | - ' fi' 2026-03-20 16:26:25.678254 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.678258 | controller | - ' ' 2026-03-20 16:26:25.678263 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.678267 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.678273 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.678277 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.678282 | controller | --timeout=300s' 2026-03-20 16:26:25.678286 | controller | - ' ' 2026-03-20 16:26:25.678290 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.678294 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.678299 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.678304 | controller | - ' ' 2026-03-20 16:26:25.678312 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.678317 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.678322 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.678327 | controller | - ' ' 2026-03-20 16:26:25.678331 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.678336 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.678340 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.678344 | controller | available"' 2026-03-20 16:26:25.678348 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.678353 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.678357 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.678362 | controller | - ' exit 1' 2026-03-20 16:26:25.678367 | controller | - ' fi' 2026-03-20 16:26:25.678373 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.678378 | controller | - ' ' 2026-03-20 16:26:25.678384 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.678388 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.678392 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.678397 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.678401 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.678405 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.678409 | controller | -- \' 2026-03-20 16:26:25.678413 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.678427 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.678432 | controller | - ' logger.go:42: 16:07:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.678436 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.678440 | controller | - ' logger.go:42: 16:07:38 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.678444 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.678449 | controller | found' 2026-03-20 16:26:25.678453 | controller | - ' logger.go:42: 16:07:39 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.678457 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.678461 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.678468 | controller | - ' ' 2026-03-20 16:26:25.678472 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.678476 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.678480 | controller | - ' ' 2026-03-20 16:26:25.678484 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.678489 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.678493 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.678500 | controller | - ' exit 1' 2026-03-20 16:26:25.678505 | controller | - ' fi' 2026-03-20 16:26:25.678509 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.678513 | controller | - ' ' 2026-03-20 16:26:25.678518 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.678522 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.678526 | controller | | base64 -d)' 2026-03-20 16:26:25.678530 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.678534 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.678539 | controller | - ' exit 1' 2026-03-20 16:26:25.678543 | controller | - ' fi' 2026-03-20 16:26:25.678547 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.678551 | controller | - ' ' 2026-03-20 16:26:25.678555 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.678560 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.678564 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.678568 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.678572 | controller | --timeout=300s' 2026-03-20 16:26:25.678576 | controller | - ' ' 2026-03-20 16:26:25.678581 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.678585 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.678589 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.678593 | controller | - ' ' 2026-03-20 16:26:25.678597 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.678602 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.678606 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.678610 | controller | - ' ' 2026-03-20 16:26:25.678614 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.678618 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.678624 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.678628 | controller | available"' 2026-03-20 16:26:25.678633 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.678637 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.678641 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.678645 | controller | - ' exit 1' 2026-03-20 16:26:25.678649 | controller | - ' fi' 2026-03-20 16:26:25.678654 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.678658 | controller | - ' ' 2026-03-20 16:26:25.678662 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.678668 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.678672 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.678677 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.678681 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.678685 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.678689 | controller | -- \' 2026-03-20 16:26:25.678721 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.678726 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.678736 | controller | - ' logger.go:42: 16:07:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.678741 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.678745 | controller | - ' logger.go:42: 16:07:39 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.678749 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.678754 | controller | found' 2026-03-20 16:26:25.678758 | controller | - ' logger.go:42: 16:07:40 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.678762 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.678766 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.678770 | controller | - ' ' 2026-03-20 16:26:25.678775 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.678779 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.678783 | controller | - ' ' 2026-03-20 16:26:25.678787 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.678791 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.678795 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.678800 | controller | - ' exit 1' 2026-03-20 16:26:25.678804 | controller | - ' fi' 2026-03-20 16:26:25.678808 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.678812 | controller | - ' ' 2026-03-20 16:26:25.678816 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.678821 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.678825 | controller | | base64 -d)' 2026-03-20 16:26:25.678829 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.678833 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.678837 | controller | - ' exit 1' 2026-03-20 16:26:25.678841 | controller | - ' fi' 2026-03-20 16:26:25.678846 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.678850 | controller | - ' ' 2026-03-20 16:26:25.678854 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.678858 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.678862 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.678866 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.678870 | controller | --timeout=300s' 2026-03-20 16:26:25.678875 | controller | - ' ' 2026-03-20 16:26:25.678879 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.678883 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.678892 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.678896 | controller | - ' ' 2026-03-20 16:26:25.678900 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.678905 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.678909 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.678913 | controller | - ' ' 2026-03-20 16:26:25.678917 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.678921 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.678925 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.678930 | controller | available"' 2026-03-20 16:26:25.678934 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.678949 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.678953 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.678958 | controller | - ' exit 1' 2026-03-20 16:26:25.678962 | controller | - ' fi' 2026-03-20 16:26:25.678966 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.678970 | controller | - ' ' 2026-03-20 16:26:25.678975 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.678979 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.678983 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.678987 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.678991 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.678995 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.679000 | controller | -- \' 2026-03-20 16:26:25.679004 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.679008 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.679022 | controller | - ' logger.go:42: 16:07:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.679027 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.679031 | controller | - ' logger.go:42: 16:07:40 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.679035 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.679039 | controller | found' 2026-03-20 16:26:25.679044 | controller | - ' logger.go:42: 16:07:41 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.679048 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.679052 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.679056 | controller | - ' ' 2026-03-20 16:26:25.679061 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.679065 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.679069 | controller | - ' ' 2026-03-20 16:26:25.679073 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.679077 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.679081 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.679096 | controller | - ' exit 1' 2026-03-20 16:26:25.679101 | controller | - ' fi' 2026-03-20 16:26:25.679105 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.679109 | controller | - ' ' 2026-03-20 16:26:25.679113 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.679118 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.679122 | controller | | base64 -d)' 2026-03-20 16:26:25.679126 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.679130 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.679134 | controller | - ' exit 1' 2026-03-20 16:26:25.679139 | controller | - ' fi' 2026-03-20 16:26:25.679143 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.679147 | controller | - ' ' 2026-03-20 16:26:25.679151 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.679155 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.679159 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.679164 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.679168 | controller | --timeout=300s' 2026-03-20 16:26:25.679172 | controller | - ' ' 2026-03-20 16:26:25.679176 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.679180 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.679185 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.679189 | controller | - ' ' 2026-03-20 16:26:25.679193 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.679200 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.679204 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.679208 | controller | - ' ' 2026-03-20 16:26:25.679213 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.679217 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.679221 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.679225 | controller | available"' 2026-03-20 16:26:25.679230 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.679235 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.679240 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.679244 | controller | - ' exit 1' 2026-03-20 16:26:25.679248 | controller | - ' fi' 2026-03-20 16:26:25.679252 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.679256 | controller | - ' ' 2026-03-20 16:26:25.679261 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.679265 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.679269 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.679273 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.679277 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.679283 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.679287 | controller | -- \' 2026-03-20 16:26:25.679291 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.679295 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.679300 | controller | - ' logger.go:42: 16:07:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.679312 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.679317 | controller | - ' logger.go:42: 16:07:41 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.679321 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.679325 | controller | found' 2026-03-20 16:26:25.679330 | controller | - ' logger.go:42: 16:07:42 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.679334 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.679338 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.679344 | controller | - ' ' 2026-03-20 16:26:25.679349 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.679353 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.679357 | controller | - ' ' 2026-03-20 16:26:25.679362 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.679366 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.679370 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.679375 | controller | - ' exit 1' 2026-03-20 16:26:25.679379 | controller | - ' fi' 2026-03-20 16:26:25.679383 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.679387 | controller | - ' ' 2026-03-20 16:26:25.679392 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.679396 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.679400 | controller | | base64 -d)' 2026-03-20 16:26:25.679404 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.679408 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.679413 | controller | - ' exit 1' 2026-03-20 16:26:25.679419 | controller | - ' fi' 2026-03-20 16:26:25.679423 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.679427 | controller | - ' ' 2026-03-20 16:26:25.679431 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.679436 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.679440 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.679444 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.679448 | controller | --timeout=300s' 2026-03-20 16:26:25.679452 | controller | - ' ' 2026-03-20 16:26:25.679457 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.679461 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.679465 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.679469 | controller | - ' ' 2026-03-20 16:26:25.679473 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.679478 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.679482 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.679489 | controller | - ' ' 2026-03-20 16:26:25.679493 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.679498 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.679502 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.679506 | controller | available"' 2026-03-20 16:26:25.679510 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.679515 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.679519 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.679523 | controller | - ' exit 1' 2026-03-20 16:26:25.679527 | controller | - ' fi' 2026-03-20 16:26:25.679531 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.679536 | controller | - ' ' 2026-03-20 16:26:25.679540 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.679544 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.679548 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.679553 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.679557 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.679561 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.679565 | controller | -- \' 2026-03-20 16:26:25.679569 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.679575 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.679595 | controller | - ' logger.go:42: 16:07:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.679600 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.679614 | controller | - ' logger.go:42: 16:07:42 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.679619 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.679624 | controller | found' 2026-03-20 16:26:25.679629 | controller | - ' logger.go:42: 16:07:43 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.679634 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.679639 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.679644 | controller | - ' ' 2026-03-20 16:26:25.679649 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.679654 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.679659 | controller | - ' ' 2026-03-20 16:26:25.679664 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.679669 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.679674 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.679678 | controller | - ' exit 1' 2026-03-20 16:26:25.679683 | controller | - ' fi' 2026-03-20 16:26:25.679688 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.679706 | controller | - ' ' 2026-03-20 16:26:25.679713 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.679718 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.679723 | controller | | base64 -d)' 2026-03-20 16:26:25.679728 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.679736 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.679741 | controller | - ' exit 1' 2026-03-20 16:26:25.679746 | controller | - ' fi' 2026-03-20 16:26:25.679751 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.679756 | controller | - ' ' 2026-03-20 16:26:25.679761 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.679766 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.679771 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.679776 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.679780 | controller | --timeout=300s' 2026-03-20 16:26:25.679785 | controller | - ' ' 2026-03-20 16:26:25.679790 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.679795 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.679800 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.679805 | controller | - ' ' 2026-03-20 16:26:25.679810 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.679815 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.679820 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.679825 | controller | - ' ' 2026-03-20 16:26:25.679830 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.679834 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.679841 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.679845 | controller | available"' 2026-03-20 16:26:25.679849 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.679854 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.679858 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.679862 | controller | - ' exit 1' 2026-03-20 16:26:25.679866 | controller | - ' fi' 2026-03-20 16:26:25.679870 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.679875 | controller | - ' ' 2026-03-20 16:26:25.679879 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.679883 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.679889 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.679894 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.679898 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.679902 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.679906 | controller | -- \' 2026-03-20 16:26:25.679910 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.679915 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.679919 | controller | - ' logger.go:42: 16:07:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.679923 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.679938 | controller | - ' logger.go:42: 16:07:44 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.679942 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.679947 | controller | found' 2026-03-20 16:26:25.679951 | controller | - ' logger.go:42: 16:07:45 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.679955 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.679959 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.679963 | controller | - ' ' 2026-03-20 16:26:25.679968 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.679972 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.679976 | controller | - ' ' 2026-03-20 16:26:25.679980 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.679985 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.679989 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.679993 | controller | - ' exit 1' 2026-03-20 16:26:25.679997 | controller | - ' fi' 2026-03-20 16:26:25.680001 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.680006 | controller | - ' ' 2026-03-20 16:26:25.680010 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.680014 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.680018 | controller | | base64 -d)' 2026-03-20 16:26:25.680023 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.680027 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.680031 | controller | - ' exit 1' 2026-03-20 16:26:25.680037 | controller | - ' fi' 2026-03-20 16:26:25.680041 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.680045 | controller | - ' ' 2026-03-20 16:26:25.680049 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.680053 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.680058 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.680062 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.680066 | controller | --timeout=300s' 2026-03-20 16:26:25.680070 | controller | - ' ' 2026-03-20 16:26:25.680075 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.680079 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.680083 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.680087 | controller | - ' ' 2026-03-20 16:26:25.680091 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.680096 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.680100 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.680104 | controller | - ' ' 2026-03-20 16:26:25.680108 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.680114 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.680118 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.680122 | controller | available"' 2026-03-20 16:26:25.680127 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.680134 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.680139 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.680143 | controller | - ' exit 1' 2026-03-20 16:26:25.680147 | controller | - ' fi' 2026-03-20 16:26:25.680151 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.680155 | controller | - ' ' 2026-03-20 16:26:25.680160 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.680164 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.680168 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.680172 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.680176 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.680181 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.680185 | controller | -- \' 2026-03-20 16:26:25.680189 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.680193 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.680197 | controller | - ' logger.go:42: 16:07:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.680201 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.680206 | controller | - ' logger.go:42: 16:07:45 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.680218 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.680223 | controller | found' 2026-03-20 16:26:25.680227 | controller | - ' logger.go:42: 16:07:46 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.680231 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.680235 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.680240 | controller | - ' ' 2026-03-20 16:26:25.680244 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.680248 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.680252 | controller | - ' ' 2026-03-20 16:26:25.680257 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.680261 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.680265 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.680269 | controller | - ' exit 1' 2026-03-20 16:26:25.680273 | controller | - ' fi' 2026-03-20 16:26:25.680278 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.680282 | controller | - ' ' 2026-03-20 16:26:25.680288 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.680293 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.680297 | controller | | base64 -d)' 2026-03-20 16:26:25.680301 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.680305 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.680309 | controller | - ' exit 1' 2026-03-20 16:26:25.680314 | controller | - ' fi' 2026-03-20 16:26:25.680318 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.680322 | controller | - ' ' 2026-03-20 16:26:25.680326 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.680331 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.680336 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.680341 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.680345 | controller | --timeout=300s' 2026-03-20 16:26:25.680349 | controller | - ' ' 2026-03-20 16:26:25.680353 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.680358 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.680362 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.680366 | controller | - ' ' 2026-03-20 16:26:25.680370 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.680374 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.680379 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.680383 | controller | - ' ' 2026-03-20 16:26:25.680387 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.680391 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.680395 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.680399 | controller | available"' 2026-03-20 16:26:25.680404 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.680408 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.680412 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.680416 | controller | - ' exit 1' 2026-03-20 16:26:25.680421 | controller | - ' fi' 2026-03-20 16:26:25.680425 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.680429 | controller | - ' ' 2026-03-20 16:26:25.680433 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.680437 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.680442 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.680447 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.680452 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.680456 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.680460 | controller | -- \' 2026-03-20 16:26:25.680464 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.680469 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.680473 | controller | - ' logger.go:42: 16:07:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.680477 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.680482 | controller | - ' logger.go:42: 16:07:46 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.680486 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.680498 | controller | found' 2026-03-20 16:26:25.680502 | controller | - ' logger.go:42: 16:07:47 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.680507 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.680511 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.680515 | controller | - ' ' 2026-03-20 16:26:25.680521 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.680525 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.680530 | controller | - ' ' 2026-03-20 16:26:25.680537 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.680541 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.680545 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.680549 | controller | - ' exit 1' 2026-03-20 16:26:25.680554 | controller | - ' fi' 2026-03-20 16:26:25.680558 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.680562 | controller | - ' ' 2026-03-20 16:26:25.680566 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.680570 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.680575 | controller | | base64 -d)' 2026-03-20 16:26:25.680579 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.680583 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.680587 | controller | - ' exit 1' 2026-03-20 16:26:25.680591 | controller | - ' fi' 2026-03-20 16:26:25.680596 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.680600 | controller | - ' ' 2026-03-20 16:26:25.680604 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.680608 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.680612 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.680617 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.680621 | controller | --timeout=300s' 2026-03-20 16:26:25.680625 | controller | - ' ' 2026-03-20 16:26:25.680629 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.680633 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.680637 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.680642 | controller | - ' ' 2026-03-20 16:26:25.680646 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.680650 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.680654 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.680658 | controller | - ' ' 2026-03-20 16:26:25.680663 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.680667 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.680671 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.680675 | controller | available"' 2026-03-20 16:26:25.680679 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.680684 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.680688 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.680705 | controller | - ' exit 1' 2026-03-20 16:26:25.680711 | controller | - ' fi' 2026-03-20 16:26:25.680716 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.680720 | controller | - ' ' 2026-03-20 16:26:25.680724 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.680730 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.680734 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.680739 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.680743 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.680747 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.680751 | controller | -- \' 2026-03-20 16:26:25.680755 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.680760 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.680764 | controller | - ' logger.go:42: 16:07:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.680768 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.680772 | controller | - ' logger.go:42: 16:07:47 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.680776 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.680780 | controller | found' 2026-03-20 16:26:25.680794 | controller | - ' logger.go:42: 16:07:48 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.680799 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.680803 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.680807 | controller | - ' ' 2026-03-20 16:26:25.680811 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.680815 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.680820 | controller | - ' ' 2026-03-20 16:26:25.680824 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.680828 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.680832 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.680837 | controller | - ' exit 1' 2026-03-20 16:26:25.680841 | controller | - ' fi' 2026-03-20 16:26:25.680845 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.680849 | controller | - ' ' 2026-03-20 16:26:25.680853 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.680857 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.680862 | controller | | base64 -d)' 2026-03-20 16:26:25.680866 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.680870 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.680874 | controller | - ' exit 1' 2026-03-20 16:26:25.680879 | controller | - ' fi' 2026-03-20 16:26:25.680883 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.680887 | controller | - ' ' 2026-03-20 16:26:25.680891 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.680895 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.680899 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.680904 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.680908 | controller | --timeout=300s' 2026-03-20 16:26:25.680912 | controller | - ' ' 2026-03-20 16:26:25.680916 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.680920 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.680927 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.680931 | controller | - ' ' 2026-03-20 16:26:25.680935 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.680939 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.680943 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.680947 | controller | - ' ' 2026-03-20 16:26:25.680952 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.680956 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.680960 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.680964 | controller | available"' 2026-03-20 16:26:25.680968 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.680972 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.680977 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.680981 | controller | - ' exit 1' 2026-03-20 16:26:25.680985 | controller | - ' fi' 2026-03-20 16:26:25.680989 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.680993 | controller | - ' ' 2026-03-20 16:26:25.680998 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.681002 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.681006 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.681010 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.681014 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.681019 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.681023 | controller | -- \' 2026-03-20 16:26:25.681027 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.681034 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.681039 | controller | - ' logger.go:42: 16:07:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.681043 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.681047 | controller | - ' logger.go:42: 16:07:48 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.681051 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.681056 | controller | found' 2026-03-20 16:26:25.681061 | controller | - ' logger.go:42: 16:07:49 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.681073 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.681078 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.681082 | controller | - ' ' 2026-03-20 16:26:25.681086 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.681091 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.681095 | controller | - ' ' 2026-03-20 16:26:25.681099 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.681104 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.681108 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.681112 | controller | - ' exit 1' 2026-03-20 16:26:25.681118 | controller | - ' fi' 2026-03-20 16:26:25.681123 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.681127 | controller | - ' ' 2026-03-20 16:26:25.681131 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.681135 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.681139 | controller | | base64 -d)' 2026-03-20 16:26:25.681144 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.681148 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.681152 | controller | - ' exit 1' 2026-03-20 16:26:25.681156 | controller | - ' fi' 2026-03-20 16:26:25.681161 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.681167 | controller | - ' ' 2026-03-20 16:26:25.681171 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.681175 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.681180 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.681184 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.681188 | controller | --timeout=300s' 2026-03-20 16:26:25.681192 | controller | - ' ' 2026-03-20 16:26:25.681196 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.681200 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.681206 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.681211 | controller | - ' ' 2026-03-20 16:26:25.681215 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.681219 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.681223 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.681227 | controller | - ' ' 2026-03-20 16:26:25.681232 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.681236 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.681240 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.681244 | controller | available"' 2026-03-20 16:26:25.681249 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.681253 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.681257 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.681261 | controller | - ' exit 1' 2026-03-20 16:26:25.681265 | controller | - ' fi' 2026-03-20 16:26:25.681270 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.681274 | controller | - ' ' 2026-03-20 16:26:25.681278 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.681282 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.681286 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.681291 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.681295 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.681300 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.681308 | controller | -- \' 2026-03-20 16:26:25.681312 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.681317 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.681323 | controller | - ' logger.go:42: 16:07:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.681327 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.681332 | controller | - ' logger.go:42: 16:07:49 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.681336 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.681340 | controller | found' 2026-03-20 16:26:25.681344 | controller | - ' logger.go:42: 16:07:50 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.681349 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.681353 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.681365 | controller | - ' ' 2026-03-20 16:26:25.681369 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.681374 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.681378 | controller | - ' ' 2026-03-20 16:26:25.681382 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.681386 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.681391 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.681395 | controller | - ' exit 1' 2026-03-20 16:26:25.681399 | controller | - ' fi' 2026-03-20 16:26:25.681403 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.681408 | controller | - ' ' 2026-03-20 16:26:25.681412 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.681416 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.681420 | controller | | base64 -d)' 2026-03-20 16:26:25.681424 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.681428 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.681433 | controller | - ' exit 1' 2026-03-20 16:26:25.681437 | controller | - ' fi' 2026-03-20 16:26:25.681441 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.681445 | controller | - ' ' 2026-03-20 16:26:25.681449 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.681454 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.681458 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.681462 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.681466 | controller | --timeout=300s' 2026-03-20 16:26:25.681472 | controller | - ' ' 2026-03-20 16:26:25.681477 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.681481 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.681485 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.681489 | controller | - ' ' 2026-03-20 16:26:25.681493 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.681498 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.681502 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.681508 | controller | - ' ' 2026-03-20 16:26:25.681512 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.681516 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.681520 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.681525 | controller | available"' 2026-03-20 16:26:25.681529 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.681533 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.681540 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.681545 | controller | - ' exit 1' 2026-03-20 16:26:25.681550 | controller | - ' fi' 2026-03-20 16:26:25.681555 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.681559 | controller | - ' ' 2026-03-20 16:26:25.681563 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.681567 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.681573 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.681577 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.681581 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.681586 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.681590 | controller | -- \' 2026-03-20 16:26:25.681594 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.681598 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.681602 | controller | - ' logger.go:42: 16:07:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.681607 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.681611 | controller | - ' logger.go:42: 16:07:50 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.681615 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.681619 | controller | found' 2026-03-20 16:26:25.681624 | controller | - ' logger.go:42: 16:07:51 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.681628 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.681634 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.681638 | controller | - ' ' 2026-03-20 16:26:25.681649 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.681655 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.681659 | controller | - ' ' 2026-03-20 16:26:25.681663 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.681667 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.681672 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.681676 | controller | - ' exit 1' 2026-03-20 16:26:25.681680 | controller | - ' fi' 2026-03-20 16:26:25.681684 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.681689 | controller | - ' ' 2026-03-20 16:26:25.681710 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.681716 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.681721 | controller | | base64 -d)' 2026-03-20 16:26:25.681725 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.681729 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.681735 | controller | - ' exit 1' 2026-03-20 16:26:25.681739 | controller | - ' fi' 2026-03-20 16:26:25.681744 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.681748 | controller | - ' ' 2026-03-20 16:26:25.681752 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.681756 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.681760 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.681765 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.681769 | controller | --timeout=300s' 2026-03-20 16:26:25.681773 | controller | - ' ' 2026-03-20 16:26:25.681777 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.681781 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.681786 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.681790 | controller | - ' ' 2026-03-20 16:26:25.681794 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.681798 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.681802 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.681807 | controller | - ' ' 2026-03-20 16:26:25.681811 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.681815 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.681819 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.681824 | controller | available"' 2026-03-20 16:26:25.681828 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.681832 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.681836 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.681840 | controller | - ' exit 1' 2026-03-20 16:26:25.681845 | controller | - ' fi' 2026-03-20 16:26:25.681849 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.681853 | controller | - ' ' 2026-03-20 16:26:25.681857 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.681861 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.681866 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.681870 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.681874 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.681878 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.681882 | controller | -- \' 2026-03-20 16:26:25.681886 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.681891 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.681895 | controller | - ' logger.go:42: 16:07:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.681899 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.681903 | controller | - ' logger.go:42: 16:07:52 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.681909 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.681913 | controller | found' 2026-03-20 16:26:25.681917 | controller | - ' logger.go:42: 16:07:53 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.681922 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.681926 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.681930 | controller | - ' ' 2026-03-20 16:26:25.681934 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.681946 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.681951 | controller | - ' ' 2026-03-20 16:26:25.681955 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.681960 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.681964 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.681968 | controller | - ' exit 1' 2026-03-20 16:26:25.681973 | controller | - ' fi' 2026-03-20 16:26:25.681977 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.681981 | controller | - ' ' 2026-03-20 16:26:25.681985 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.681989 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.681994 | controller | | base64 -d)' 2026-03-20 16:26:25.681998 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.682002 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.682006 | controller | - ' exit 1' 2026-03-20 16:26:25.682010 | controller | - ' fi' 2026-03-20 16:26:25.682015 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.682019 | controller | - ' ' 2026-03-20 16:26:25.682023 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.682027 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.682031 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.682035 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.682040 | controller | --timeout=300s' 2026-03-20 16:26:25.682044 | controller | - ' ' 2026-03-20 16:26:25.682048 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.682052 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.682056 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.682061 | controller | - ' ' 2026-03-20 16:26:25.682067 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.682072 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.682078 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.682083 | controller | - ' ' 2026-03-20 16:26:25.682089 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.682094 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.682100 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.682104 | controller | available"' 2026-03-20 16:26:25.682108 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.682112 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.682119 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.682123 | controller | - ' exit 1' 2026-03-20 16:26:25.682127 | controller | - ' fi' 2026-03-20 16:26:25.682132 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.682136 | controller | - ' ' 2026-03-20 16:26:25.682140 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.682144 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.682148 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.682152 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.682164 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.682169 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.682173 | controller | -- \' 2026-03-20 16:26:25.682177 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.682182 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.682186 | controller | - ' logger.go:42: 16:07:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.682190 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.682194 | controller | - ' logger.go:42: 16:07:53 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.682200 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.682204 | controller | found' 2026-03-20 16:26:25.682209 | controller | - ' logger.go:42: 16:07:54 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.682213 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.682217 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.682222 | controller | - ' ' 2026-03-20 16:26:25.682226 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.682239 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.682244 | controller | - ' ' 2026-03-20 16:26:25.682248 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.682252 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.682256 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.682261 | controller | - ' exit 1' 2026-03-20 16:26:25.682265 | controller | - ' fi' 2026-03-20 16:26:25.682270 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.682274 | controller | - ' ' 2026-03-20 16:26:25.682278 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.682283 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.682287 | controller | | base64 -d)' 2026-03-20 16:26:25.682291 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.682295 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.682299 | controller | - ' exit 1' 2026-03-20 16:26:25.682304 | controller | - ' fi' 2026-03-20 16:26:25.682308 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.682312 | controller | - ' ' 2026-03-20 16:26:25.682318 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.682324 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.682329 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.682337 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.682341 | controller | --timeout=300s' 2026-03-20 16:26:25.682346 | controller | - ' ' 2026-03-20 16:26:25.682350 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.682354 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.682358 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.682363 | controller | - ' ' 2026-03-20 16:26:25.682367 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.682371 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.682375 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.682379 | controller | - ' ' 2026-03-20 16:26:25.682384 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.682388 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.682392 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.682396 | controller | available"' 2026-03-20 16:26:25.682400 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.682405 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.682409 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.682413 | controller | - ' exit 1' 2026-03-20 16:26:25.682417 | controller | - ' fi' 2026-03-20 16:26:25.682421 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.682426 | controller | - ' ' 2026-03-20 16:26:25.682430 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.682434 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.682438 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.682442 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.682446 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.682451 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.682455 | controller | -- \' 2026-03-20 16:26:25.682460 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.682465 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.682471 | controller | - ' logger.go:42: 16:07:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.682477 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.682482 | controller | - ' logger.go:42: 16:07:54 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.682488 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.682493 | controller | found' 2026-03-20 16:26:25.682499 | controller | - ' logger.go:42: 16:07:55 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.682503 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.682507 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.682512 | controller | - ' ' 2026-03-20 16:26:25.682516 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.682523 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.682535 | controller | - ' ' 2026-03-20 16:26:25.682541 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.682545 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.682549 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.682556 | controller | - ' exit 1' 2026-03-20 16:26:25.682560 | controller | - ' fi' 2026-03-20 16:26:25.682564 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.682569 | controller | - ' ' 2026-03-20 16:26:25.682573 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.682578 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.682583 | controller | | base64 -d)' 2026-03-20 16:26:25.682588 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.682594 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.682599 | controller | - ' exit 1' 2026-03-20 16:26:25.682605 | controller | - ' fi' 2026-03-20 16:26:25.682609 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.682613 | controller | - ' ' 2026-03-20 16:26:25.682619 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.682624 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.682628 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.682632 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.682636 | controller | --timeout=300s' 2026-03-20 16:26:25.682641 | controller | - ' ' 2026-03-20 16:26:25.682645 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.682649 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.682653 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.682657 | controller | - ' ' 2026-03-20 16:26:25.682662 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.682666 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.682670 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.682674 | controller | - ' ' 2026-03-20 16:26:25.682678 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.682682 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.682687 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.682691 | controller | available"' 2026-03-20 16:26:25.682711 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.682715 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.682719 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.682723 | controller | - ' exit 1' 2026-03-20 16:26:25.682727 | controller | - ' fi' 2026-03-20 16:26:25.682732 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.682736 | controller | - ' ' 2026-03-20 16:26:25.682740 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.682744 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.682752 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.682758 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.682764 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.682770 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.682776 | controller | -- \' 2026-03-20 16:26:25.682780 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.682784 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.682788 | controller | - ' logger.go:42: 16:07:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.682792 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.682797 | controller | - ' logger.go:42: 16:07:55 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.682801 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.682805 | controller | found' 2026-03-20 16:26:25.682813 | controller | - ' logger.go:42: 16:07:56 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.682818 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.682824 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.682830 | controller | - ' ' 2026-03-20 16:26:25.682835 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.682840 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.682844 | controller | - ' ' 2026-03-20 16:26:25.682858 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.682862 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.682867 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.682871 | controller | - ' exit 1' 2026-03-20 16:26:25.682875 | controller | - ' fi' 2026-03-20 16:26:25.682880 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.682884 | controller | - ' ' 2026-03-20 16:26:25.682892 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.682896 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.682900 | controller | | base64 -d)' 2026-03-20 16:26:25.682904 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.682909 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.682913 | controller | - ' exit 1' 2026-03-20 16:26:25.682917 | controller | - ' fi' 2026-03-20 16:26:25.682921 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.682925 | controller | - ' ' 2026-03-20 16:26:25.682930 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.682934 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.682938 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.682942 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.682946 | controller | --timeout=300s' 2026-03-20 16:26:25.682951 | controller | - ' ' 2026-03-20 16:26:25.682955 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.682959 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.682963 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.682969 | controller | - ' ' 2026-03-20 16:26:25.682974 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.682978 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.682982 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.682986 | controller | - ' ' 2026-03-20 16:26:25.682991 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.682995 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.682999 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.683003 | controller | available"' 2026-03-20 16:26:25.683008 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.683012 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.683016 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.683020 | controller | - ' exit 1' 2026-03-20 16:26:25.683024 | controller | - ' fi' 2026-03-20 16:26:25.683029 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.683033 | controller | - ' ' 2026-03-20 16:26:25.683037 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.683041 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.683045 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.683049 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.683054 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.683059 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.683065 | controller | -- \' 2026-03-20 16:26:25.683071 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.683076 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.683081 | controller | - ' logger.go:42: 16:07:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.683085 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.683091 | controller | - ' logger.go:42: 16:07:56 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.683096 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.683100 | controller | found' 2026-03-20 16:26:25.683104 | controller | - ' logger.go:42: 16:07:57 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.683108 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.683113 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.683117 | controller | - ' ' 2026-03-20 16:26:25.683121 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.683125 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.683130 | controller | - ' ' 2026-03-20 16:26:25.683134 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.683146 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.683151 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.683155 | controller | - ' exit 1' 2026-03-20 16:26:25.683159 | controller | - ' fi' 2026-03-20 16:26:25.683165 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.683170 | controller | - ' ' 2026-03-20 16:26:25.683174 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.683178 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.683182 | controller | | base64 -d)' 2026-03-20 16:26:25.683186 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.683191 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.683195 | controller | - ' exit 1' 2026-03-20 16:26:25.683199 | controller | - ' fi' 2026-03-20 16:26:25.683203 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.683207 | controller | - ' ' 2026-03-20 16:26:25.683212 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.683216 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.683220 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.683224 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.683228 | controller | --timeout=300s' 2026-03-20 16:26:25.683233 | controller | - ' ' 2026-03-20 16:26:25.683239 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.683244 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.683249 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.683254 | controller | - ' ' 2026-03-20 16:26:25.683260 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.683269 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.683274 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.683280 | controller | - ' ' 2026-03-20 16:26:25.683284 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.683289 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.683293 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.683297 | controller | available"' 2026-03-20 16:26:25.683301 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.683306 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.683310 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.683314 | controller | - ' exit 1' 2026-03-20 16:26:25.683318 | controller | - ' fi' 2026-03-20 16:26:25.683322 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.683327 | controller | - ' ' 2026-03-20 16:26:25.683331 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.683335 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.683339 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.683343 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.683348 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.683352 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.683356 | controller | -- \' 2026-03-20 16:26:25.683362 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.683366 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.683371 | controller | - ' logger.go:42: 16:07:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.683375 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.683379 | controller | - ' logger.go:42: 16:07:57 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.683383 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.683387 | controller | found' 2026-03-20 16:26:25.683392 | controller | - ' logger.go:42: 16:07:58 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.683396 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.683400 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.683404 | controller | - ' ' 2026-03-20 16:26:25.683410 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.683415 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.683419 | controller | - ' ' 2026-03-20 16:26:25.683423 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.683427 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.683440 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.683445 | controller | - ' exit 1' 2026-03-20 16:26:25.683449 | controller | - ' fi' 2026-03-20 16:26:25.683453 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.683457 | controller | - ' ' 2026-03-20 16:26:25.683462 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.683466 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.683470 | controller | | base64 -d)' 2026-03-20 16:26:25.683474 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.683478 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.683483 | controller | - ' exit 1' 2026-03-20 16:26:25.683487 | controller | - ' fi' 2026-03-20 16:26:25.683491 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.683495 | controller | - ' ' 2026-03-20 16:26:25.683500 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.683504 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.683508 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.683512 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.683516 | controller | --timeout=300s' 2026-03-20 16:26:25.683521 | controller | - ' ' 2026-03-20 16:26:25.683525 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.683529 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.683533 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.683538 | controller | - ' ' 2026-03-20 16:26:25.683542 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.683546 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.683550 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.683554 | controller | - ' ' 2026-03-20 16:26:25.683560 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.683564 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.683569 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.683573 | controller | available"' 2026-03-20 16:26:25.683578 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.683611 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.683617 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.683622 | controller | - ' exit 1' 2026-03-20 16:26:25.683628 | controller | - ' fi' 2026-03-20 16:26:25.683632 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.683636 | controller | - ' ' 2026-03-20 16:26:25.683640 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.683645 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.683649 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.683653 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.683657 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.683661 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.683665 | controller | -- \' 2026-03-20 16:26:25.683669 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.683674 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.683678 | controller | - ' logger.go:42: 16:07:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.683682 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.683686 | controller | - ' logger.go:42: 16:07:58 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.683690 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.683710 | controller | found' 2026-03-20 16:26:25.683715 | controller | - ' logger.go:42: 16:07:59 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.683719 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.683723 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.683727 | controller | - ' ' 2026-03-20 16:26:25.683731 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.683736 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.683740 | controller | - ' ' 2026-03-20 16:26:25.683744 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.683748 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.683752 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.683765 | controller | - ' exit 1' 2026-03-20 16:26:25.683770 | controller | - ' fi' 2026-03-20 16:26:25.683778 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.683783 | controller | - ' ' 2026-03-20 16:26:25.683787 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.683791 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.683795 | controller | | base64 -d)' 2026-03-20 16:26:25.683799 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.683804 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.683810 | controller | - ' exit 1' 2026-03-20 16:26:25.683814 | controller | - ' fi' 2026-03-20 16:26:25.683818 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.683823 | controller | - ' ' 2026-03-20 16:26:25.683827 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.683831 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.683835 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.683839 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.683843 | controller | --timeout=300s' 2026-03-20 16:26:25.683848 | controller | - ' ' 2026-03-20 16:26:25.683852 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.683856 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.683860 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.683864 | controller | - ' ' 2026-03-20 16:26:25.683868 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.683873 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.683877 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.683881 | controller | - ' ' 2026-03-20 16:26:25.683885 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.683889 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.683895 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.683899 | controller | available"' 2026-03-20 16:26:25.683904 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.683908 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.683912 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.683916 | controller | - ' exit 1' 2026-03-20 16:26:25.683920 | controller | - ' fi' 2026-03-20 16:26:25.683925 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.683929 | controller | - ' ' 2026-03-20 16:26:25.683933 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.683937 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.683941 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.683945 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.683949 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.683954 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.683958 | controller | -- \' 2026-03-20 16:26:25.683962 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.683966 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.683971 | controller | - ' logger.go:42: 16:07:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.683975 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.683979 | controller | - ' logger.go:42: 16:08:00 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.683985 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.683989 | controller | found' 2026-03-20 16:26:25.683993 | controller | - ' logger.go:42: 16:08:01 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.683997 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.684001 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.684006 | controller | - ' ' 2026-03-20 16:26:25.684012 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.684017 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.684022 | controller | - ' ' 2026-03-20 16:26:25.684026 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.684030 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.684034 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.684038 | controller | - ' exit 1' 2026-03-20 16:26:25.684043 | controller | - ' fi' 2026-03-20 16:26:25.684055 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.684060 | controller | - ' ' 2026-03-20 16:26:25.684064 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.684068 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.684073 | controller | | base64 -d)' 2026-03-20 16:26:25.684077 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.684081 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.684085 | controller | - ' exit 1' 2026-03-20 16:26:25.684090 | controller | - ' fi' 2026-03-20 16:26:25.684094 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.684098 | controller | - ' ' 2026-03-20 16:26:25.684102 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.684107 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.684111 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.684115 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.684119 | controller | --timeout=300s' 2026-03-20 16:26:25.684123 | controller | - ' ' 2026-03-20 16:26:25.684127 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.684132 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.684136 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.684140 | controller | - ' ' 2026-03-20 16:26:25.684144 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.684148 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.684153 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.684157 | controller | - ' ' 2026-03-20 16:26:25.684161 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.684165 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.684169 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.684174 | controller | available"' 2026-03-20 16:26:25.684178 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.684184 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.684188 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.684195 | controller | - ' exit 1' 2026-03-20 16:26:25.684200 | controller | - ' fi' 2026-03-20 16:26:25.684204 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.684208 | controller | - ' ' 2026-03-20 16:26:25.684212 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.684216 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.684221 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.684225 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.684229 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.684233 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.684239 | controller | -- \' 2026-03-20 16:26:25.684243 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.684248 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.684252 | controller | - ' logger.go:42: 16:08:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.684256 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.684260 | controller | - ' logger.go:42: 16:08:01 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.684264 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.684268 | controller | found' 2026-03-20 16:26:25.684273 | controller | - ' logger.go:42: 16:08:02 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.684277 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.684281 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.684285 | controller | - ' ' 2026-03-20 16:26:25.684290 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.684294 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.684299 | controller | - ' ' 2026-03-20 16:26:25.684304 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.684309 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.684313 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.684318 | controller | - ' exit 1' 2026-03-20 16:26:25.684324 | controller | - ' fi' 2026-03-20 16:26:25.684328 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.684340 | controller | - ' ' 2026-03-20 16:26:25.684344 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.684348 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.684353 | controller | | base64 -d)' 2026-03-20 16:26:25.684357 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.684361 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.684365 | controller | - ' exit 1' 2026-03-20 16:26:25.684370 | controller | - ' fi' 2026-03-20 16:26:25.684374 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.684378 | controller | - ' ' 2026-03-20 16:26:25.684382 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.684386 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.684391 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.684395 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.684401 | controller | --timeout=300s' 2026-03-20 16:26:25.684405 | controller | - ' ' 2026-03-20 16:26:25.684410 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.684414 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.684418 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.684422 | controller | - ' ' 2026-03-20 16:26:25.684426 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.684431 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.684435 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.684439 | controller | - ' ' 2026-03-20 16:26:25.684443 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.684447 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.684451 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.684456 | controller | available"' 2026-03-20 16:26:25.684460 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.684464 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.684468 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.684472 | controller | - ' exit 1' 2026-03-20 16:26:25.684477 | controller | - ' fi' 2026-03-20 16:26:25.684481 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.684485 | controller | - ' ' 2026-03-20 16:26:25.684489 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.684493 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.684497 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.684502 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.684506 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.684510 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.684514 | controller | -- \' 2026-03-20 16:26:25.684518 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.684522 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.684527 | controller | - ' logger.go:42: 16:08:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.684531 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.684535 | controller | - ' logger.go:42: 16:08:02 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.684539 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.684543 | controller | found' 2026-03-20 16:26:25.684548 | controller | - ' logger.go:42: 16:08:03 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.684552 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.684556 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.684560 | controller | - ' ' 2026-03-20 16:26:25.684564 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.684568 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.684574 | controller | - ' ' 2026-03-20 16:26:25.684580 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.684584 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.684589 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.684593 | controller | - ' exit 1' 2026-03-20 16:26:25.684597 | controller | - ' fi' 2026-03-20 16:26:25.684601 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.684606 | controller | - ' ' 2026-03-20 16:26:25.684617 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.684622 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.684626 | controller | | base64 -d)' 2026-03-20 16:26:25.684630 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.684635 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.684639 | controller | - ' exit 1' 2026-03-20 16:26:25.684643 | controller | - ' fi' 2026-03-20 16:26:25.684647 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.684651 | controller | - ' ' 2026-03-20 16:26:25.684656 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.684660 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.684664 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.684668 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.684672 | controller | --timeout=300s' 2026-03-20 16:26:25.684676 | controller | - ' ' 2026-03-20 16:26:25.684681 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.684685 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.684689 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.684706 | controller | - ' ' 2026-03-20 16:26:25.684710 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.684715 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.684719 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.684723 | controller | - ' ' 2026-03-20 16:26:25.684727 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.684731 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.684735 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.684740 | controller | available"' 2026-03-20 16:26:25.684744 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.684748 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.684752 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.684756 | controller | - ' exit 1' 2026-03-20 16:26:25.684761 | controller | - ' fi' 2026-03-20 16:26:25.684768 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.684772 | controller | - ' ' 2026-03-20 16:26:25.684776 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.684781 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.684785 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.684791 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.684795 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.684799 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.684804 | controller | -- \' 2026-03-20 16:26:25.684808 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.684812 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.684816 | controller | - ' logger.go:42: 16:08:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.684820 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.684824 | controller | - ' logger.go:42: 16:08:03 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.684829 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.684833 | controller | found' 2026-03-20 16:26:25.684837 | controller | - ' logger.go:42: 16:08:04 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.684841 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.684845 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.684850 | controller | - ' ' 2026-03-20 16:26:25.684854 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.684858 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.684862 | controller | - ' ' 2026-03-20 16:26:25.684866 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.684870 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.684875 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.684879 | controller | - ' exit 1' 2026-03-20 16:26:25.684883 | controller | - ' fi' 2026-03-20 16:26:25.684887 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.684891 | controller | - ' ' 2026-03-20 16:26:25.684896 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.684907 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.684912 | controller | | base64 -d)' 2026-03-20 16:26:25.684917 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.684921 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.684926 | controller | - ' exit 1' 2026-03-20 16:26:25.684930 | controller | - ' fi' 2026-03-20 16:26:25.684934 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.684938 | controller | - ' ' 2026-03-20 16:26:25.684942 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.684946 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.684951 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.684955 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.684959 | controller | --timeout=300s' 2026-03-20 16:26:25.684963 | controller | - ' ' 2026-03-20 16:26:25.684967 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.684972 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.684976 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.684982 | controller | - ' ' 2026-03-20 16:26:25.684986 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.684990 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.684995 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.684999 | controller | - ' ' 2026-03-20 16:26:25.685003 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.685007 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.685011 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.685016 | controller | available"' 2026-03-20 16:26:25.685020 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.685024 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.685028 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.685032 | controller | - ' exit 1' 2026-03-20 16:26:25.685037 | controller | - ' fi' 2026-03-20 16:26:25.685041 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.685045 | controller | - ' ' 2026-03-20 16:26:25.685049 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.685054 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.685058 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.685064 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.685069 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.685075 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.685081 | controller | -- \' 2026-03-20 16:26:25.685086 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.685092 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.685098 | controller | - ' logger.go:42: 16:08:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.685102 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.685107 | controller | - ' logger.go:42: 16:08:04 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.685111 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.685115 | controller | found' 2026-03-20 16:26:25.685119 | controller | - ' logger.go:42: 16:08:05 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.685123 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.685128 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.685132 | controller | - ' ' 2026-03-20 16:26:25.685136 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.685140 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.685144 | controller | - ' ' 2026-03-20 16:26:25.685149 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.685153 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.685157 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.685161 | controller | - ' exit 1' 2026-03-20 16:26:25.685165 | controller | - ' fi' 2026-03-20 16:26:25.685169 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.685178 | controller | - ' ' 2026-03-20 16:26:25.685182 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.685196 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.685200 | controller | | base64 -d)' 2026-03-20 16:26:25.685205 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.685209 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.685213 | controller | - ' exit 1' 2026-03-20 16:26:25.685217 | controller | - ' fi' 2026-03-20 16:26:25.685222 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.685226 | controller | - ' ' 2026-03-20 16:26:25.685230 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.685234 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.685238 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.685243 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.685247 | controller | --timeout=300s' 2026-03-20 16:26:25.685251 | controller | - ' ' 2026-03-20 16:26:25.685257 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.685261 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.685265 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.685269 | controller | - ' ' 2026-03-20 16:26:25.685274 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.685278 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.685283 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.685288 | controller | - ' ' 2026-03-20 16:26:25.685292 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.685296 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.685300 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.685304 | controller | available"' 2026-03-20 16:26:25.685309 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.685313 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.685317 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.685321 | controller | - ' exit 1' 2026-03-20 16:26:25.685325 | controller | - ' fi' 2026-03-20 16:26:25.685329 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.685334 | controller | - ' ' 2026-03-20 16:26:25.685338 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.685343 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.685349 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.685354 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.685360 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.685366 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.685371 | controller | -- \' 2026-03-20 16:26:25.685376 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.685383 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.685387 | controller | - ' logger.go:42: 16:08:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.685391 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.685396 | controller | - ' logger.go:42: 16:08:05 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.685400 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.685404 | controller | found' 2026-03-20 16:26:25.685408 | controller | - ' logger.go:42: 16:08:06 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.685413 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.685417 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.685421 | controller | - ' ' 2026-03-20 16:26:25.685425 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.685429 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.685434 | controller | - ' ' 2026-03-20 16:26:25.685438 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.685442 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.685446 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.685450 | controller | - ' exit 1' 2026-03-20 16:26:25.685455 | controller | - ' fi' 2026-03-20 16:26:25.685459 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.685463 | controller | - ' ' 2026-03-20 16:26:25.685469 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.685473 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.685478 | controller | | base64 -d)' 2026-03-20 16:26:25.685490 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.685495 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.685499 | controller | - ' exit 1' 2026-03-20 16:26:25.685503 | controller | - ' fi' 2026-03-20 16:26:25.685508 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.685512 | controller | - ' ' 2026-03-20 16:26:25.685516 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.685520 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.685525 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.685529 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.685533 | controller | --timeout=300s' 2026-03-20 16:26:25.685539 | controller | - ' ' 2026-03-20 16:26:25.685543 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.685547 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.685552 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.685556 | controller | - ' ' 2026-03-20 16:26:25.685560 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.685564 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.685568 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.685573 | controller | - ' ' 2026-03-20 16:26:25.685577 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.685586 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.685591 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.685597 | controller | available"' 2026-03-20 16:26:25.685602 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.685608 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.685614 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.685619 | controller | - ' exit 1' 2026-03-20 16:26:25.685625 | controller | - ' fi' 2026-03-20 16:26:25.685629 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.685633 | controller | - ' ' 2026-03-20 16:26:25.685638 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.685642 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.685646 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.685650 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.685654 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.685659 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.685663 | controller | -- \' 2026-03-20 16:26:25.685667 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.685671 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.685676 | controller | - ' logger.go:42: 16:08:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.685680 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.685684 | controller | - ' logger.go:42: 16:08:07 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.685688 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.685705 | controller | found' 2026-03-20 16:26:25.685712 | controller | - ' logger.go:42: 16:08:08 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.685716 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.685720 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.685724 | controller | - ' ' 2026-03-20 16:26:25.685729 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.685733 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.685737 | controller | - ' ' 2026-03-20 16:26:25.685741 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.685745 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.685749 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.685754 | controller | - ' exit 1' 2026-03-20 16:26:25.685758 | controller | - ' fi' 2026-03-20 16:26:25.685762 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.685766 | controller | - ' ' 2026-03-20 16:26:25.685771 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.685775 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.685779 | controller | | base64 -d)' 2026-03-20 16:26:25.685783 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.685796 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.685801 | controller | - ' exit 1' 2026-03-20 16:26:25.685807 | controller | - ' fi' 2026-03-20 16:26:25.685812 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.685816 | controller | - ' ' 2026-03-20 16:26:25.685820 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.685824 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.685831 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.685836 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.685840 | controller | --timeout=300s' 2026-03-20 16:26:25.685844 | controller | - ' ' 2026-03-20 16:26:25.685852 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.685857 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.685863 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.685869 | controller | - ' ' 2026-03-20 16:26:25.685874 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.685880 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.685886 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.685890 | controller | - ' ' 2026-03-20 16:26:25.685894 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.685898 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.685903 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.685907 | controller | available"' 2026-03-20 16:26:25.685911 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.685915 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.685919 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.685923 | controller | - ' exit 1' 2026-03-20 16:26:25.685928 | controller | - ' fi' 2026-03-20 16:26:25.685932 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.685936 | controller | - ' ' 2026-03-20 16:26:25.685940 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.685944 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.685949 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.685953 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.685957 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.685961 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.685965 | controller | -- \' 2026-03-20 16:26:25.685970 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.685974 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.685978 | controller | - ' logger.go:42: 16:08:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.685982 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.685986 | controller | - ' logger.go:42: 16:08:08 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.685990 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.685997 | controller | found' 2026-03-20 16:26:25.686001 | controller | - ' logger.go:42: 16:08:09 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.686006 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.686010 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.686014 | controller | - ' ' 2026-03-20 16:26:25.686018 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.686022 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.686027 | controller | - ' ' 2026-03-20 16:26:25.686031 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.686035 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.686039 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.686043 | controller | - ' exit 1' 2026-03-20 16:26:25.686048 | controller | - ' fi' 2026-03-20 16:26:25.686054 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.686059 | controller | - ' ' 2026-03-20 16:26:25.686065 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.686070 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.686076 | controller | | base64 -d)' 2026-03-20 16:26:25.686081 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.686094 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.686099 | controller | - ' exit 1' 2026-03-20 16:26:25.686103 | controller | - ' fi' 2026-03-20 16:26:25.686108 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.686112 | controller | - ' ' 2026-03-20 16:26:25.686116 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.686120 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.686125 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.686129 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.686133 | controller | --timeout=300s' 2026-03-20 16:26:25.686137 | controller | - ' ' 2026-03-20 16:26:25.686141 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.686145 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.686149 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.686154 | controller | - ' ' 2026-03-20 16:26:25.686158 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.686162 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.686166 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.686170 | controller | - ' ' 2026-03-20 16:26:25.686176 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.686181 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.686186 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.686192 | controller | available"' 2026-03-20 16:26:25.686197 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.686201 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.686205 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.686213 | controller | - ' exit 1' 2026-03-20 16:26:25.686217 | controller | - ' fi' 2026-03-20 16:26:25.686222 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.686226 | controller | - ' ' 2026-03-20 16:26:25.686230 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.686234 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.686238 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.686242 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.686247 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.686251 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.686255 | controller | -- \' 2026-03-20 16:26:25.686259 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.686263 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.686270 | controller | - ' logger.go:42: 16:08:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.686274 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.686279 | controller | - ' logger.go:42: 16:08:09 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.686283 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.686287 | controller | found' 2026-03-20 16:26:25.686291 | controller | - ' logger.go:42: 16:08:10 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.686295 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.686300 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.686304 | controller | - ' ' 2026-03-20 16:26:25.686308 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.686312 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.686317 | controller | - ' ' 2026-03-20 16:26:25.686321 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.686325 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.686329 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.686333 | controller | - ' exit 1' 2026-03-20 16:26:25.686338 | controller | - ' fi' 2026-03-20 16:26:25.686342 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.686346 | controller | - ' ' 2026-03-20 16:26:25.686350 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.686354 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.686359 | controller | | base64 -d)' 2026-03-20 16:26:25.686363 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.686367 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.686379 | controller | - ' exit 1' 2026-03-20 16:26:25.686386 | controller | - ' fi' 2026-03-20 16:26:25.686390 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.686394 | controller | - ' ' 2026-03-20 16:26:25.686398 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.686402 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.686407 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.686411 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.686419 | controller | --timeout=300s' 2026-03-20 16:26:25.686423 | controller | - ' ' 2026-03-20 16:26:25.686427 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.686431 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.686435 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.686439 | controller | - ' ' 2026-03-20 16:26:25.686444 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.686448 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.686452 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.686456 | controller | - ' ' 2026-03-20 16:26:25.686460 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.686465 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.686469 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.686473 | controller | available"' 2026-03-20 16:26:25.686477 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.686481 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.686485 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.686490 | controller | - ' exit 1' 2026-03-20 16:26:25.686494 | controller | - ' fi' 2026-03-20 16:26:25.686498 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.686502 | controller | - ' ' 2026-03-20 16:26:25.686506 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.686510 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.686515 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.686519 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.686523 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.686527 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.686531 | controller | -- \' 2026-03-20 16:26:25.686535 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.686540 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.686544 | controller | - ' logger.go:42: 16:08:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.686548 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.686552 | controller | - ' logger.go:42: 16:08:10 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.686556 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.686560 | controller | found' 2026-03-20 16:26:25.686565 | controller | - ' logger.go:42: 16:08:11 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.686569 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.686573 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.686579 | controller | - ' ' 2026-03-20 16:26:25.686584 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.686588 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.686594 | controller | - ' ' 2026-03-20 16:26:25.686598 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.686603 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.686607 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.686611 | controller | - ' exit 1' 2026-03-20 16:26:25.686616 | controller | - ' fi' 2026-03-20 16:26:25.686620 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.686624 | controller | - ' ' 2026-03-20 16:26:25.686628 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.686634 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.686639 | controller | | base64 -d)' 2026-03-20 16:26:25.686643 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.686647 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.686651 | controller | - ' exit 1' 2026-03-20 16:26:25.686656 | controller | - ' fi' 2026-03-20 16:26:25.686667 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.686672 | controller | - ' ' 2026-03-20 16:26:25.686676 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.686680 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.686685 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.686689 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.686705 | controller | --timeout=300s' 2026-03-20 16:26:25.686712 | controller | - ' ' 2026-03-20 16:26:25.686716 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.686720 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.686724 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.686729 | controller | - ' ' 2026-03-20 16:26:25.686733 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.686737 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.686741 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.686746 | controller | - ' ' 2026-03-20 16:26:25.686750 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.686754 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.686758 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.686762 | controller | available"' 2026-03-20 16:26:25.686766 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.686773 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.686778 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.686782 | controller | - ' exit 1' 2026-03-20 16:26:25.686786 | controller | - ' fi' 2026-03-20 16:26:25.686791 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.686795 | controller | - ' ' 2026-03-20 16:26:25.686799 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.686803 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.686808 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.686814 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.686818 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.686822 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.686827 | controller | -- \' 2026-03-20 16:26:25.686831 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.686835 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.686839 | controller | - ' logger.go:42: 16:08:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.686843 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.686848 | controller | - ' logger.go:42: 16:08:11 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.686852 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.686856 | controller | found' 2026-03-20 16:26:25.686860 | controller | - ' logger.go:42: 16:08:12 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.686864 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.686869 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.686873 | controller | - ' ' 2026-03-20 16:26:25.686877 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.686881 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.686885 | controller | - ' ' 2026-03-20 16:26:25.686890 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.686894 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.686898 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.686902 | controller | - ' exit 1' 2026-03-20 16:26:25.686906 | controller | - ' fi' 2026-03-20 16:26:25.686911 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.686915 | controller | - ' ' 2026-03-20 16:26:25.686919 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.686923 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.686928 | controller | | base64 -d)' 2026-03-20 16:26:25.686935 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.686940 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.686944 | controller | - ' exit 1' 2026-03-20 16:26:25.686948 | controller | - ' fi' 2026-03-20 16:26:25.686952 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.686965 | controller | - ' ' 2026-03-20 16:26:25.686970 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.686974 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.686978 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.686982 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.686986 | controller | --timeout=300s' 2026-03-20 16:26:25.686991 | controller | - ' ' 2026-03-20 16:26:25.686995 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.686999 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.687003 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.687008 | controller | - ' ' 2026-03-20 16:26:25.687015 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.687019 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.687023 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.687027 | controller | - ' ' 2026-03-20 16:26:25.687033 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.687037 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.687042 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.687046 | controller | available"' 2026-03-20 16:26:25.687050 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.687054 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.687058 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.687063 | controller | - ' exit 1' 2026-03-20 16:26:25.687067 | controller | - ' fi' 2026-03-20 16:26:25.687071 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.687075 | controller | - ' ' 2026-03-20 16:26:25.687079 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.687083 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.687088 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.687092 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.687096 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.687100 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.687104 | controller | -- \' 2026-03-20 16:26:25.687108 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.687113 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.687117 | controller | - ' logger.go:42: 16:08:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.687121 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.687125 | controller | - ' logger.go:42: 16:08:12 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.687129 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.687133 | controller | found' 2026-03-20 16:26:25.687138 | controller | - ' logger.go:42: 16:08:13 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.687142 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.687146 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.687150 | controller | - ' ' 2026-03-20 16:26:25.687154 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.687159 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.687163 | controller | - ' ' 2026-03-20 16:26:25.687167 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.687171 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.687175 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.687179 | controller | - ' exit 1' 2026-03-20 16:26:25.687184 | controller | - ' fi' 2026-03-20 16:26:25.687188 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.687192 | controller | - ' ' 2026-03-20 16:26:25.687196 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.687202 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.687207 | controller | | base64 -d)' 2026-03-20 16:26:25.687211 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.687215 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.687219 | controller | - ' exit 1' 2026-03-20 16:26:25.687223 | controller | - ' fi' 2026-03-20 16:26:25.687228 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.687232 | controller | - ' ' 2026-03-20 16:26:25.687244 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.687249 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.687253 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.687257 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.687262 | controller | --timeout=300s' 2026-03-20 16:26:25.687266 | controller | - ' ' 2026-03-20 16:26:25.687270 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.687274 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.687279 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.687283 | controller | - ' ' 2026-03-20 16:26:25.687287 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.687291 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.687295 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.687299 | controller | - ' ' 2026-03-20 16:26:25.687303 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.687310 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.687314 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.687318 | controller | available"' 2026-03-20 16:26:25.687322 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.687327 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.687331 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.687335 | controller | - ' exit 1' 2026-03-20 16:26:25.687339 | controller | - ' fi' 2026-03-20 16:26:25.687343 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.687347 | controller | - ' ' 2026-03-20 16:26:25.687352 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.687356 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.687360 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.687364 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.687368 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.687372 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.687377 | controller | -- \' 2026-03-20 16:26:25.687381 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.687387 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.687391 | controller | - ' logger.go:42: 16:08:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.687395 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.687399 | controller | - ' logger.go:42: 16:08:14 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.687403 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.687408 | controller | found' 2026-03-20 16:26:25.687412 | controller | - ' logger.go:42: 16:08:15 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.687416 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.687420 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.687424 | controller | - ' ' 2026-03-20 16:26:25.687428 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.687433 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.687437 | controller | - ' ' 2026-03-20 16:26:25.687441 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.687445 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.687449 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.687454 | controller | - ' exit 1' 2026-03-20 16:26:25.687458 | controller | - ' fi' 2026-03-20 16:26:25.687462 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.687466 | controller | - ' ' 2026-03-20 16:26:25.687470 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.687474 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.687479 | controller | | base64 -d)' 2026-03-20 16:26:25.687483 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.687487 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.687491 | controller | - ' exit 1' 2026-03-20 16:26:25.687495 | controller | - ' fi' 2026-03-20 16:26:25.687499 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.687503 | controller | - ' ' 2026-03-20 16:26:25.687508 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.687519 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.687524 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.687528 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.687533 | controller | --timeout=300s' 2026-03-20 16:26:25.687538 | controller | - ' ' 2026-03-20 16:26:25.687542 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.687546 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.687550 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.687554 | controller | - ' ' 2026-03-20 16:26:25.687559 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.687563 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.687567 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.687571 | controller | - ' ' 2026-03-20 16:26:25.687575 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.687579 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.687608 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.687614 | controller | available"' 2026-03-20 16:26:25.687619 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.687624 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.687628 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.687633 | controller | - ' exit 1' 2026-03-20 16:26:25.687638 | controller | - ' fi' 2026-03-20 16:26:25.687643 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.687648 | controller | - ' ' 2026-03-20 16:26:25.687653 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.687658 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.687663 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.687668 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.687673 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.687677 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.687682 | controller | -- \' 2026-03-20 16:26:25.687687 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.687708 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.687715 | controller | - ' logger.go:42: 16:08:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.687721 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.687726 | controller | - ' logger.go:42: 16:08:15 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.687730 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.687735 | controller | found' 2026-03-20 16:26:25.687740 | controller | - ' logger.go:42: 16:08:16 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.687745 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.687750 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.687755 | controller | - ' ' 2026-03-20 16:26:25.687760 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.687765 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.687770 | controller | - ' ' 2026-03-20 16:26:25.687774 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.687779 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.687784 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.687789 | controller | - ' exit 1' 2026-03-20 16:26:25.687794 | controller | - ' fi' 2026-03-20 16:26:25.687799 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.687804 | controller | - ' ' 2026-03-20 16:26:25.687809 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.687813 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.687818 | controller | | base64 -d)' 2026-03-20 16:26:25.687823 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.687828 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.687833 | controller | - ' exit 1' 2026-03-20 16:26:25.687838 | controller | - ' fi' 2026-03-20 16:26:25.687842 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.687849 | controller | - ' ' 2026-03-20 16:26:25.687853 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.687857 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.687870 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.687875 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.687880 | controller | --timeout=300s' 2026-03-20 16:26:25.687884 | controller | - ' ' 2026-03-20 16:26:25.687888 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.687892 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.687897 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.687901 | controller | - ' ' 2026-03-20 16:26:25.687905 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.687909 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.687913 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.687917 | controller | - ' ' 2026-03-20 16:26:25.687922 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.687926 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.687932 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.687936 | controller | available"' 2026-03-20 16:26:25.687941 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.687945 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.687949 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.687953 | controller | - ' exit 1' 2026-03-20 16:26:25.687958 | controller | - ' fi' 2026-03-20 16:26:25.687962 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.687966 | controller | - ' ' 2026-03-20 16:26:25.687970 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.687974 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.687979 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.687983 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.687987 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.687991 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.687995 | controller | -- \' 2026-03-20 16:26:25.688000 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.688004 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.688008 | controller | - ' logger.go:42: 16:08:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.688012 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.688016 | controller | - ' logger.go:42: 16:08:16 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.688021 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.688025 | controller | found' 2026-03-20 16:26:25.688031 | controller | - ' logger.go:42: 16:08:17 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.688035 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.688039 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.688043 | controller | - ' ' 2026-03-20 16:26:25.688050 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.688054 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.688059 | controller | - ' ' 2026-03-20 16:26:25.688063 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.688067 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.688071 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.688076 | controller | - ' exit 1' 2026-03-20 16:26:25.688080 | controller | - ' fi' 2026-03-20 16:26:25.688084 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.688088 | controller | - ' ' 2026-03-20 16:26:25.688094 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.688098 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.688103 | controller | | base64 -d)' 2026-03-20 16:26:25.688107 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.688111 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.688115 | controller | - ' exit 1' 2026-03-20 16:26:25.688119 | controller | - ' fi' 2026-03-20 16:26:25.688124 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.688128 | controller | - ' ' 2026-03-20 16:26:25.688132 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.688136 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.688148 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.688153 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.688157 | controller | --timeout=300s' 2026-03-20 16:26:25.688161 | controller | - ' ' 2026-03-20 16:26:25.688166 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.688170 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.688174 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.688178 | controller | - ' ' 2026-03-20 16:26:25.688183 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.688187 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.688191 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.688195 | controller | - ' ' 2026-03-20 16:26:25.688201 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.688205 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.688209 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.688213 | controller | available"' 2026-03-20 16:26:25.688217 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.688224 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.688228 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.688232 | controller | - ' exit 1' 2026-03-20 16:26:25.688238 | controller | - ' fi' 2026-03-20 16:26:25.688242 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.688247 | controller | - ' ' 2026-03-20 16:26:25.688251 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.688255 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.688259 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.688263 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.688268 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.688272 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.688276 | controller | -- \' 2026-03-20 16:26:25.688280 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.688284 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.688289 | controller | - ' logger.go:42: 16:08:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.688293 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.688299 | controller | - ' logger.go:42: 16:08:17 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.688304 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.688308 | controller | found' 2026-03-20 16:26:25.688325 | controller | - ' logger.go:42: 16:08:18 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.688330 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.688334 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.688338 | controller | - ' ' 2026-03-20 16:26:25.688342 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.688347 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.688351 | controller | - ' ' 2026-03-20 16:26:25.688355 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.688359 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.688363 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.688368 | controller | - ' exit 1' 2026-03-20 16:26:25.688372 | controller | - ' fi' 2026-03-20 16:26:25.688376 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.688380 | controller | - ' ' 2026-03-20 16:26:25.688384 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.688389 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.688393 | controller | | base64 -d)' 2026-03-20 16:26:25.688397 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.688401 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.688405 | controller | - ' exit 1' 2026-03-20 16:26:25.688410 | controller | - ' fi' 2026-03-20 16:26:25.688414 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.688418 | controller | - ' ' 2026-03-20 16:26:25.688422 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.688426 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.688431 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.688442 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.688447 | controller | --timeout=300s' 2026-03-20 16:26:25.688456 | controller | - ' ' 2026-03-20 16:26:25.688460 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.688464 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.688471 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.688475 | controller | - ' ' 2026-03-20 16:26:25.688480 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.688484 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.688488 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.688501 | controller | - ' ' 2026-03-20 16:26:25.688505 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.688509 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.688514 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.688518 | controller | available"' 2026-03-20 16:26:25.688522 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.688526 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.688530 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.688535 | controller | - ' exit 1' 2026-03-20 16:26:25.688539 | controller | - ' fi' 2026-03-20 16:26:25.688543 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.688547 | controller | - ' ' 2026-03-20 16:26:25.688551 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.688555 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.688559 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.688564 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.688568 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.688572 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.688576 | controller | -- \' 2026-03-20 16:26:25.688581 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.688585 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.688589 | controller | - ' logger.go:42: 16:08:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.688593 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.688597 | controller | - ' logger.go:42: 16:08:18 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.688602 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.688613 | controller | found' 2026-03-20 16:26:25.688618 | controller | - ' logger.go:42: 16:08:19 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.688622 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.688626 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.688630 | controller | - ' ' 2026-03-20 16:26:25.688635 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.688639 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.688643 | controller | - ' ' 2026-03-20 16:26:25.688647 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.688662 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.688667 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.688671 | controller | - ' exit 1' 2026-03-20 16:26:25.688676 | controller | - ' fi' 2026-03-20 16:26:25.688680 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.688684 | controller | - ' ' 2026-03-20 16:26:25.688688 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.688703 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.688710 | controller | | base64 -d)' 2026-03-20 16:26:25.688714 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.688718 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.688722 | controller | - ' exit 1' 2026-03-20 16:26:25.688727 | controller | - ' fi' 2026-03-20 16:26:25.688731 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.688735 | controller | - ' ' 2026-03-20 16:26:25.688739 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.688743 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.688748 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.688767 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.688772 | controller | --timeout=300s' 2026-03-20 16:26:25.688776 | controller | - ' ' 2026-03-20 16:26:25.688780 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.688785 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.688789 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.688793 | controller | - ' ' 2026-03-20 16:26:25.688797 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.688801 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.688806 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.688810 | controller | - ' ' 2026-03-20 16:26:25.688814 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.688818 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.688822 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.688827 | controller | available"' 2026-03-20 16:26:25.688831 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.688835 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.688839 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.688843 | controller | - ' exit 1' 2026-03-20 16:26:25.688847 | controller | - ' fi' 2026-03-20 16:26:25.688863 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.688868 | controller | - ' ' 2026-03-20 16:26:25.688874 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.688878 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.688882 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.688886 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.688899 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.688904 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.688908 | controller | -- \' 2026-03-20 16:26:25.688912 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.688916 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.688921 | controller | - ' logger.go:42: 16:08:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.688925 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.688929 | controller | - ' logger.go:42: 16:08:19 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.688933 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.688937 | controller | found' 2026-03-20 16:26:25.688942 | controller | - ' logger.go:42: 16:08:20 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.688946 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.688950 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.688954 | controller | - ' ' 2026-03-20 16:26:25.688958 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.688963 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.688967 | controller | - ' ' 2026-03-20 16:26:25.688971 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.688975 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.688979 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.688984 | controller | - ' exit 1' 2026-03-20 16:26:25.688988 | controller | - ' fi' 2026-03-20 16:26:25.688992 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.688996 | controller | - ' ' 2026-03-20 16:26:25.689000 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.689005 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.689009 | controller | | base64 -d)' 2026-03-20 16:26:25.689013 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.689017 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.689021 | controller | - ' exit 1' 2026-03-20 16:26:25.689026 | controller | - ' fi' 2026-03-20 16:26:25.689030 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.689034 | controller | - ' ' 2026-03-20 16:26:25.689038 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.689042 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.689046 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.689051 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.689055 | controller | --timeout=300s' 2026-03-20 16:26:25.689067 | controller | - ' ' 2026-03-20 16:26:25.689072 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.689076 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.689080 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.689084 | controller | - ' ' 2026-03-20 16:26:25.689088 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.689095 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.689099 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.689103 | controller | - ' ' 2026-03-20 16:26:25.689110 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.689122 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.689127 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.689131 | controller | available"' 2026-03-20 16:26:25.689135 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.689140 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.689144 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.689148 | controller | - ' exit 1' 2026-03-20 16:26:25.689152 | controller | - ' fi' 2026-03-20 16:26:25.689157 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.689161 | controller | - ' ' 2026-03-20 16:26:25.689165 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.689169 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.689173 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.689177 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.689182 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.689186 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.689190 | controller | -- \' 2026-03-20 16:26:25.689194 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.689198 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.689203 | controller | - ' logger.go:42: 16:08:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.689207 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.689211 | controller | - ' logger.go:42: 16:08:20 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.689215 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.689219 | controller | found' 2026-03-20 16:26:25.689224 | controller | - ' logger.go:42: 16:08:21 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.689228 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.689232 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.689236 | controller | - ' ' 2026-03-20 16:26:25.689240 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.689244 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.689249 | controller | - ' ' 2026-03-20 16:26:25.689253 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.689257 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.689261 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.689265 | controller | - ' exit 1' 2026-03-20 16:26:25.689270 | controller | - ' fi' 2026-03-20 16:26:25.689274 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.689278 | controller | - ' ' 2026-03-20 16:26:25.689282 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.689288 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.689292 | controller | | base64 -d)' 2026-03-20 16:26:25.689296 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.689301 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.689305 | controller | - ' exit 1' 2026-03-20 16:26:25.689309 | controller | - ' fi' 2026-03-20 16:26:25.689313 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.689319 | controller | - ' ' 2026-03-20 16:26:25.689323 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.689327 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.689332 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.689336 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.689340 | controller | --timeout=300s' 2026-03-20 16:26:25.689344 | controller | - ' ' 2026-03-20 16:26:25.689356 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.689361 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.689365 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.689369 | controller | - ' ' 2026-03-20 16:26:25.689374 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.689378 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.689382 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.689387 | controller | - ' ' 2026-03-20 16:26:25.689391 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.689395 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.689399 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.689403 | controller | available"' 2026-03-20 16:26:25.689408 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.689414 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.689418 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.689422 | controller | - ' exit 1' 2026-03-20 16:26:25.689426 | controller | - ' fi' 2026-03-20 16:26:25.689431 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.689435 | controller | - ' ' 2026-03-20 16:26:25.689439 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.689443 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.689447 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.689453 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.689459 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.689464 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.689468 | controller | -- \' 2026-03-20 16:26:25.689472 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.689476 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.689483 | controller | - ' logger.go:42: 16:08:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.689487 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.689491 | controller | - ' logger.go:42: 16:08:22 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.689495 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.689499 | controller | found' 2026-03-20 16:26:25.689504 | controller | - ' logger.go:42: 16:08:23 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.689508 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.689512 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.689516 | controller | - ' ' 2026-03-20 16:26:25.689520 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.689525 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.689529 | controller | - ' ' 2026-03-20 16:26:25.689533 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.689537 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.689542 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.689546 | controller | - ' exit 1' 2026-03-20 16:26:25.689550 | controller | - ' fi' 2026-03-20 16:26:25.689554 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.689558 | controller | - ' ' 2026-03-20 16:26:25.689563 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.689567 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.689571 | controller | | base64 -d)' 2026-03-20 16:26:25.689583 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.689588 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.689592 | controller | - ' exit 1' 2026-03-20 16:26:25.689596 | controller | - ' fi' 2026-03-20 16:26:25.689601 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.689605 | controller | - ' ' 2026-03-20 16:26:25.689609 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.689613 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.689618 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.689622 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.689626 | controller | --timeout=300s' 2026-03-20 16:26:25.689634 | controller | - ' ' 2026-03-20 16:26:25.689646 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.689651 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.689655 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.689659 | controller | - ' ' 2026-03-20 16:26:25.689664 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.689668 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.689672 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.689676 | controller | - ' ' 2026-03-20 16:26:25.689681 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.689685 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.689691 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.689711 | controller | available"' 2026-03-20 16:26:25.689715 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.689719 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.689723 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.689728 | controller | - ' exit 1' 2026-03-20 16:26:25.689732 | controller | - ' fi' 2026-03-20 16:26:25.689736 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.689740 | controller | - ' ' 2026-03-20 16:26:25.689744 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.689749 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.689753 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.689757 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.689761 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.689765 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.689770 | controller | -- \' 2026-03-20 16:26:25.689774 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.689778 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.689782 | controller | - ' logger.go:42: 16:08:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.689786 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.689791 | controller | - ' logger.go:42: 16:08:23 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.689795 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.689799 | controller | found' 2026-03-20 16:26:25.689803 | controller | - ' logger.go:42: 16:08:24 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.689807 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.689812 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.689816 | controller | - ' ' 2026-03-20 16:26:25.689820 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.689824 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.689828 | controller | - ' ' 2026-03-20 16:26:25.689833 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.689837 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.689841 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.689845 | controller | - ' exit 1' 2026-03-20 16:26:25.689849 | controller | - ' fi' 2026-03-20 16:26:25.689854 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.689858 | controller | - ' ' 2026-03-20 16:26:25.689862 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.689866 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.689870 | controller | | base64 -d)' 2026-03-20 16:26:25.689875 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.689879 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.689883 | controller | - ' exit 1' 2026-03-20 16:26:25.689887 | controller | - ' fi' 2026-03-20 16:26:25.689891 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.689898 | controller | - ' ' 2026-03-20 16:26:25.689902 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.689906 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.689910 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.689914 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.689919 | controller | --timeout=300s' 2026-03-20 16:26:25.689923 | controller | - ' ' 2026-03-20 16:26:25.689927 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.689939 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.689945 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.689949 | controller | - ' ' 2026-03-20 16:26:25.689953 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.689957 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.689961 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.689966 | controller | - ' ' 2026-03-20 16:26:25.689970 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.689974 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.689978 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.689982 | controller | available"' 2026-03-20 16:26:25.689987 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.689991 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.689995 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.689999 | controller | - ' exit 1' 2026-03-20 16:26:25.690004 | controller | - ' fi' 2026-03-20 16:26:25.690008 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.690012 | controller | - ' ' 2026-03-20 16:26:25.690016 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.690020 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.690025 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.690029 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.690033 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.690037 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.690041 | controller | -- \' 2026-03-20 16:26:25.690045 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.690052 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.690056 | controller | - ' logger.go:42: 16:08:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.690060 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.690065 | controller | - ' logger.go:42: 16:08:24 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.690069 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.690073 | controller | found' 2026-03-20 16:26:25.690077 | controller | - ' logger.go:42: 16:08:25 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.690083 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.690087 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.690091 | controller | - ' ' 2026-03-20 16:26:25.690095 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.690100 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.690104 | controller | - ' ' 2026-03-20 16:26:25.690108 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.690112 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.690116 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.690121 | controller | - ' exit 1' 2026-03-20 16:26:25.690125 | controller | - ' fi' 2026-03-20 16:26:25.690129 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.690133 | controller | - ' ' 2026-03-20 16:26:25.690137 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.690142 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.690146 | controller | | base64 -d)' 2026-03-20 16:26:25.690150 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.690154 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.690158 | controller | - ' exit 1' 2026-03-20 16:26:25.690163 | controller | - ' fi' 2026-03-20 16:26:25.690169 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.690173 | controller | - ' ' 2026-03-20 16:26:25.690177 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.690181 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.690186 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.690190 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.690194 | controller | --timeout=300s' 2026-03-20 16:26:25.690198 | controller | - ' ' 2026-03-20 16:26:25.690202 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.690213 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.690218 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.690222 | controller | - ' ' 2026-03-20 16:26:25.690227 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.690231 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.690235 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.690239 | controller | - ' ' 2026-03-20 16:26:25.690243 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.690248 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.690252 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.690256 | controller | available"' 2026-03-20 16:26:25.690260 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.690265 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.690269 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.690273 | controller | - ' exit 1' 2026-03-20 16:26:25.690277 | controller | - ' fi' 2026-03-20 16:26:25.690281 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.690288 | controller | - ' ' 2026-03-20 16:26:25.690293 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.690297 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.690301 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.690305 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.690309 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.690314 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.690318 | controller | -- \' 2026-03-20 16:26:25.690322 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.690326 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.690331 | controller | - ' logger.go:42: 16:08:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.690335 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.690341 | controller | - ' logger.go:42: 16:08:25 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.690346 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.690350 | controller | found' 2026-03-20 16:26:25.690354 | controller | - ' logger.go:42: 16:08:26 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.690358 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.690363 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.690367 | controller | - ' ' 2026-03-20 16:26:25.690371 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.690375 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.690379 | controller | - ' ' 2026-03-20 16:26:25.690384 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.690388 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.690392 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.690396 | controller | - ' exit 1' 2026-03-20 16:26:25.690401 | controller | - ' fi' 2026-03-20 16:26:25.690405 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.690409 | controller | - ' ' 2026-03-20 16:26:25.690414 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.690418 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.690423 | controller | | base64 -d)' 2026-03-20 16:26:25.690427 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.690431 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.690435 | controller | - ' exit 1' 2026-03-20 16:26:25.690439 | controller | - ' fi' 2026-03-20 16:26:25.690444 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.690448 | controller | - ' ' 2026-03-20 16:26:25.690452 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.690456 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.690460 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.690465 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.690469 | controller | --timeout=300s' 2026-03-20 16:26:25.690473 | controller | - ' ' 2026-03-20 16:26:25.690477 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.690485 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.690496 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.690501 | controller | - ' ' 2026-03-20 16:26:25.690505 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.690511 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.690515 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.690520 | controller | - ' ' 2026-03-20 16:26:25.690524 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.690528 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.690532 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.690536 | controller | available"' 2026-03-20 16:26:25.690541 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.690545 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.690549 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.690553 | controller | - ' exit 1' 2026-03-20 16:26:25.690557 | controller | - ' fi' 2026-03-20 16:26:25.690562 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.690566 | controller | - ' ' 2026-03-20 16:26:25.690570 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.690574 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.690578 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.690582 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.690587 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.690591 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.690595 | controller | -- \' 2026-03-20 16:26:25.690599 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.690603 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.690608 | controller | - ' logger.go:42: 16:08:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.690612 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.690618 | controller | - ' logger.go:42: 16:08:26 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.690622 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.690626 | controller | found' 2026-03-20 16:26:25.690630 | controller | - ' logger.go:42: 16:08:27 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.690634 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.690638 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.690643 | controller | - ' ' 2026-03-20 16:26:25.690647 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.690651 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.690655 | controller | - ' ' 2026-03-20 16:26:25.690659 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.690666 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.690671 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.690677 | controller | - ' exit 1' 2026-03-20 16:26:25.690681 | controller | - ' fi' 2026-03-20 16:26:25.690685 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.690689 | controller | - ' ' 2026-03-20 16:26:25.690708 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.690715 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.690719 | controller | | base64 -d)' 2026-03-20 16:26:25.690724 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.690728 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.690732 | controller | - ' exit 1' 2026-03-20 16:26:25.690736 | controller | - ' fi' 2026-03-20 16:26:25.690740 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.690744 | controller | - ' ' 2026-03-20 16:26:25.690749 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.690753 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.690757 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.690761 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.690765 | controller | --timeout=300s' 2026-03-20 16:26:25.690770 | controller | - ' ' 2026-03-20 16:26:25.690774 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.690778 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.690790 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.690794 | controller | - ' ' 2026-03-20 16:26:25.690799 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.690803 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.690807 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.690811 | controller | - ' ' 2026-03-20 16:26:25.690816 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.690820 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.690824 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.690828 | controller | available"' 2026-03-20 16:26:25.690832 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.690836 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.690841 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.690845 | controller | - ' exit 1' 2026-03-20 16:26:25.690849 | controller | - ' fi' 2026-03-20 16:26:25.690853 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.690857 | controller | - ' ' 2026-03-20 16:26:25.690862 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.690866 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.690870 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.690874 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.690880 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.690884 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.690889 | controller | -- \' 2026-03-20 16:26:25.690893 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.690897 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.690901 | controller | - ' logger.go:42: 16:08:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.690906 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.690910 | controller | - ' logger.go:42: 16:08:27 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.690914 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.690918 | controller | found' 2026-03-20 16:26:25.690922 | controller | - ' logger.go:42: 16:08:28 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.690927 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.690931 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.690935 | controller | - ' ' 2026-03-20 16:26:25.690939 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.690943 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.690948 | controller | - ' ' 2026-03-20 16:26:25.690952 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.690956 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.690960 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.690965 | controller | - ' exit 1' 2026-03-20 16:26:25.690969 | controller | - ' fi' 2026-03-20 16:26:25.690973 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.690977 | controller | - ' ' 2026-03-20 16:26:25.690982 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.690986 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.690990 | controller | | base64 -d)' 2026-03-20 16:26:25.690994 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.690998 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.691003 | controller | - ' exit 1' 2026-03-20 16:26:25.691007 | controller | - ' fi' 2026-03-20 16:26:25.691011 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.691015 | controller | - ' ' 2026-03-20 16:26:25.691020 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.691024 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.691028 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.691032 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.691036 | controller | --timeout=300s' 2026-03-20 16:26:25.691040 | controller | - ' ' 2026-03-20 16:26:25.691045 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.691049 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.691053 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.691064 | controller | - ' ' 2026-03-20 16:26:25.691069 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.691073 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.691088 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.691093 | controller | - ' ' 2026-03-20 16:26:25.691098 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.691102 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.691106 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.691111 | controller | available"' 2026-03-20 16:26:25.691115 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.691119 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.691123 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.691127 | controller | - ' exit 1' 2026-03-20 16:26:25.691132 | controller | - ' fi' 2026-03-20 16:26:25.691136 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.691140 | controller | - ' ' 2026-03-20 16:26:25.691144 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.691148 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.691153 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.691157 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.691163 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.691168 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.691172 | controller | -- \' 2026-03-20 16:26:25.691176 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.691180 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.691185 | controller | - ' logger.go:42: 16:08:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.691189 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.691193 | controller | - ' logger.go:42: 16:08:28 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.691199 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.691203 | controller | found' 2026-03-20 16:26:25.691207 | controller | - ' logger.go:42: 16:08:29 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.691211 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.691216 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.691220 | controller | - ' ' 2026-03-20 16:26:25.691224 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.691228 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.691232 | controller | - ' ' 2026-03-20 16:26:25.691237 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.691241 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.691245 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.691249 | controller | - ' exit 1' 2026-03-20 16:26:25.691255 | controller | - ' fi' 2026-03-20 16:26:25.691259 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.691264 | controller | - ' ' 2026-03-20 16:26:25.691268 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.691272 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.691278 | controller | | base64 -d)' 2026-03-20 16:26:25.691282 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.691287 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.691291 | controller | - ' exit 1' 2026-03-20 16:26:25.691295 | controller | - ' fi' 2026-03-20 16:26:25.691299 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.691303 | controller | - ' ' 2026-03-20 16:26:25.691308 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.691312 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.691316 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.691320 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.691324 | controller | --timeout=300s' 2026-03-20 16:26:25.691329 | controller | - ' ' 2026-03-20 16:26:25.691333 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.691337 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.691341 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.691345 | controller | - ' ' 2026-03-20 16:26:25.691357 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.691362 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.691366 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.691370 | controller | - ' ' 2026-03-20 16:26:25.691375 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.691379 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.691383 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.691387 | controller | available"' 2026-03-20 16:26:25.691391 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.691396 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.691400 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.691404 | controller | - ' exit 1' 2026-03-20 16:26:25.691408 | controller | - ' fi' 2026-03-20 16:26:25.691412 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.691417 | controller | - ' ' 2026-03-20 16:26:25.691421 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.691425 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.691429 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.691433 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.691438 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.691442 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.691449 | controller | -- \' 2026-03-20 16:26:25.691453 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.691457 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.691463 | controller | - ' logger.go:42: 16:08:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.691467 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.691472 | controller | - ' logger.go:42: 16:08:30 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.691476 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.691480 | controller | found' 2026-03-20 16:26:25.691484 | controller | - ' logger.go:42: 16:08:31 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.691489 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.691493 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.691497 | controller | - ' ' 2026-03-20 16:26:25.691501 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.691506 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.691511 | controller | - ' ' 2026-03-20 16:26:25.691515 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.691519 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.691523 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.691527 | controller | - ' exit 1' 2026-03-20 16:26:25.691532 | controller | - ' fi' 2026-03-20 16:26:25.691536 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.691540 | controller | - ' ' 2026-03-20 16:26:25.691544 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.691548 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.691553 | controller | | base64 -d)' 2026-03-20 16:26:25.691557 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.691561 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.691565 | controller | - ' exit 1' 2026-03-20 16:26:25.691569 | controller | - ' fi' 2026-03-20 16:26:25.691574 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.691578 | controller | - ' ' 2026-03-20 16:26:25.691582 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.691587 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.691607 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.691612 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.691617 | controller | --timeout=300s' 2026-03-20 16:26:25.691622 | controller | - ' ' 2026-03-20 16:26:25.691627 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.691632 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.691637 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.691642 | controller | - ' ' 2026-03-20 16:26:25.691647 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.691659 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.691664 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.691670 | controller | - ' ' 2026-03-20 16:26:25.691675 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.691679 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.691684 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.691692 | controller | available"' 2026-03-20 16:26:25.691712 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.691717 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.691722 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.691727 | controller | - ' exit 1' 2026-03-20 16:26:25.691732 | controller | - ' fi' 2026-03-20 16:26:25.691737 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.691742 | controller | - ' ' 2026-03-20 16:26:25.691747 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.691752 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.691756 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.691761 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.691766 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.691771 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.691776 | controller | -- \' 2026-03-20 16:26:25.691781 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.691786 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.691791 | controller | - ' logger.go:42: 16:08:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.691796 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.691800 | controller | - ' logger.go:42: 16:08:31 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.691805 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.691810 | controller | found' 2026-03-20 16:26:25.691818 | controller | - ' logger.go:42: 16:08:32 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.691823 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.691828 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.691833 | controller | - ' ' 2026-03-20 16:26:25.691837 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.691843 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.691847 | controller | - ' ' 2026-03-20 16:26:25.691852 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.691856 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.691860 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.691864 | controller | - ' exit 1' 2026-03-20 16:26:25.691869 | controller | - ' fi' 2026-03-20 16:26:25.691873 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.691877 | controller | - ' ' 2026-03-20 16:26:25.691881 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.691885 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.691890 | controller | | base64 -d)' 2026-03-20 16:26:25.691894 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.691898 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.691902 | controller | - ' exit 1' 2026-03-20 16:26:25.691906 | controller | - ' fi' 2026-03-20 16:26:25.691911 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.691915 | controller | - ' ' 2026-03-20 16:26:25.691919 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.691925 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.691930 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.691934 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.691938 | controller | --timeout=300s' 2026-03-20 16:26:25.691943 | controller | - ' ' 2026-03-20 16:26:25.691947 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.691952 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.691956 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.691960 | controller | - ' ' 2026-03-20 16:26:25.691964 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.691977 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.691982 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.691986 | controller | - ' ' 2026-03-20 16:26:25.691990 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.691994 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.692001 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.692005 | controller | available"' 2026-03-20 16:26:25.692009 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.692013 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.692019 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.692023 | controller | - ' exit 1' 2026-03-20 16:26:25.692028 | controller | - ' fi' 2026-03-20 16:26:25.692033 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.692037 | controller | - ' ' 2026-03-20 16:26:25.692041 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.692045 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.692049 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.692054 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.692059 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.692063 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.692067 | controller | -- \' 2026-03-20 16:26:25.692071 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.692076 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.692080 | controller | - ' logger.go:42: 16:08:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.692084 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.692088 | controller | - ' logger.go:42: 16:08:32 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.692092 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.692097 | controller | found' 2026-03-20 16:26:25.692101 | controller | - ' logger.go:42: 16:08:33 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.692105 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.692111 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.692115 | controller | - ' ' 2026-03-20 16:26:25.692120 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.692124 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.692128 | controller | - ' ' 2026-03-20 16:26:25.692132 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.692136 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.692141 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.692145 | controller | - ' exit 1' 2026-03-20 16:26:25.692149 | controller | - ' fi' 2026-03-20 16:26:25.692153 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.692157 | controller | - ' ' 2026-03-20 16:26:25.692162 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.692166 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.692170 | controller | | base64 -d)' 2026-03-20 16:26:25.692174 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.692178 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.692183 | controller | - ' exit 1' 2026-03-20 16:26:25.692187 | controller | - ' fi' 2026-03-20 16:26:25.692191 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.692195 | controller | - ' ' 2026-03-20 16:26:25.692200 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.692204 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.692208 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.692212 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.692216 | controller | --timeout=300s' 2026-03-20 16:26:25.692221 | controller | - ' ' 2026-03-20 16:26:25.692225 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.692229 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.692233 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.692238 | controller | - ' ' 2026-03-20 16:26:25.692242 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.692254 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.692259 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.692263 | controller | - ' ' 2026-03-20 16:26:25.692267 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.692271 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.692276 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.692280 | controller | available"' 2026-03-20 16:26:25.692284 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.692288 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.692293 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.692297 | controller | - ' exit 1' 2026-03-20 16:26:25.692303 | controller | - ' fi' 2026-03-20 16:26:25.692308 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.692313 | controller | - ' ' 2026-03-20 16:26:25.692318 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.692322 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.692326 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.692330 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.692335 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.692339 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.692343 | controller | -- \' 2026-03-20 16:26:25.692347 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.692351 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.692356 | controller | - ' logger.go:42: 16:08:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.692360 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.692364 | controller | - ' logger.go:42: 16:08:33 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.692368 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.692372 | controller | found' 2026-03-20 16:26:25.692377 | controller | - ' logger.go:42: 16:08:34 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.692381 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.692385 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.692389 | controller | - ' ' 2026-03-20 16:26:25.692396 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.692401 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.692405 | controller | - ' ' 2026-03-20 16:26:25.692409 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.692413 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.692417 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.692422 | controller | - ' exit 1' 2026-03-20 16:26:25.692426 | controller | - ' fi' 2026-03-20 16:26:25.692430 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.692434 | controller | - ' ' 2026-03-20 16:26:25.692438 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.692443 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.692447 | controller | | base64 -d)' 2026-03-20 16:26:25.692451 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.692455 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.692459 | controller | - ' exit 1' 2026-03-20 16:26:25.692463 | controller | - ' fi' 2026-03-20 16:26:25.692468 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.692472 | controller | - ' ' 2026-03-20 16:26:25.692476 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.692480 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.692485 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.692489 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.692493 | controller | --timeout=300s' 2026-03-20 16:26:25.692497 | controller | - ' ' 2026-03-20 16:26:25.692501 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.692507 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.692511 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.692516 | controller | - ' ' 2026-03-20 16:26:25.692520 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.692524 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.692537 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.692542 | controller | - ' ' 2026-03-20 16:26:25.692547 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.692552 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.692556 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.692560 | controller | available"' 2026-03-20 16:26:25.692564 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.692568 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.692573 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.692577 | controller | - ' exit 1' 2026-03-20 16:26:25.692581 | controller | - ' fi' 2026-03-20 16:26:25.692585 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.692589 | controller | - ' ' 2026-03-20 16:26:25.692594 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.692598 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.692602 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.692606 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.692610 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.692614 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.692619 | controller | -- \' 2026-03-20 16:26:25.692623 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.692627 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.692631 | controller | - ' logger.go:42: 16:08:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.692635 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.692640 | controller | - ' logger.go:42: 16:08:34 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.692644 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.692648 | controller | found' 2026-03-20 16:26:25.692652 | controller | - ' logger.go:42: 16:08:35 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.692656 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.692660 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.692665 | controller | - ' ' 2026-03-20 16:26:25.692669 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.692673 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.692677 | controller | - ' ' 2026-03-20 16:26:25.692681 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.692685 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.692706 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.692713 | controller | - ' exit 1' 2026-03-20 16:26:25.692717 | controller | - ' fi' 2026-03-20 16:26:25.692721 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.692725 | controller | - ' ' 2026-03-20 16:26:25.692730 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.692734 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.692738 | controller | | base64 -d)' 2026-03-20 16:26:25.692742 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.692747 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.692751 | controller | - ' exit 1' 2026-03-20 16:26:25.692755 | controller | - ' fi' 2026-03-20 16:26:25.692760 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.692765 | controller | - ' ' 2026-03-20 16:26:25.692769 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.692773 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.692777 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.692781 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.692786 | controller | --timeout=300s' 2026-03-20 16:26:25.692790 | controller | - ' ' 2026-03-20 16:26:25.692794 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.692798 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.692802 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.692806 | controller | - ' ' 2026-03-20 16:26:25.692811 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.692815 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.692826 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.692831 | controller | - ' ' 2026-03-20 16:26:25.692835 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.692840 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.692844 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.692848 | controller | available"' 2026-03-20 16:26:25.692852 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.692856 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.692861 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.692865 | controller | - ' exit 1' 2026-03-20 16:26:25.692869 | controller | - ' fi' 2026-03-20 16:26:25.692873 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.692877 | controller | - ' ' 2026-03-20 16:26:25.692882 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.692886 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.692890 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.692894 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.692898 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.692904 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.692909 | controller | -- \' 2026-03-20 16:26:25.692913 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.692917 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.692921 | controller | - ' logger.go:42: 16:08:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.692926 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.692930 | controller | - ' logger.go:42: 16:08:35 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.692934 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.692938 | controller | found' 2026-03-20 16:26:25.692942 | controller | - ' logger.go:42: 16:08:36 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.692946 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.692951 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.692955 | controller | - ' ' 2026-03-20 16:26:25.692959 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.692966 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.692970 | controller | - ' ' 2026-03-20 16:26:25.692975 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.692979 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.692983 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.692987 | controller | - ' exit 1' 2026-03-20 16:26:25.692991 | controller | - ' fi' 2026-03-20 16:26:25.692996 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.693000 | controller | - ' ' 2026-03-20 16:26:25.693004 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.693008 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.693012 | controller | | base64 -d)' 2026-03-20 16:26:25.693016 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.693021 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.693025 | controller | - ' exit 1' 2026-03-20 16:26:25.693029 | controller | - ' fi' 2026-03-20 16:26:25.693033 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.693037 | controller | - ' ' 2026-03-20 16:26:25.693042 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.693046 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.693050 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.693054 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.693058 | controller | --timeout=300s' 2026-03-20 16:26:25.693062 | controller | - ' ' 2026-03-20 16:26:25.693067 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.693071 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.693075 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.693079 | controller | - ' ' 2026-03-20 16:26:25.693083 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.693087 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.693094 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.693098 | controller | - ' ' 2026-03-20 16:26:25.693109 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.693115 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.693119 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.693123 | controller | available"' 2026-03-20 16:26:25.693128 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.693132 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.693136 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.693140 | controller | - ' exit 1' 2026-03-20 16:26:25.693144 | controller | - ' fi' 2026-03-20 16:26:25.693149 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.693153 | controller | - ' ' 2026-03-20 16:26:25.693157 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.693161 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.693166 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.693170 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.693174 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.693178 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.693182 | controller | -- \' 2026-03-20 16:26:25.693188 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.693192 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.693197 | controller | - ' logger.go:42: 16:08:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.693201 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.693205 | controller | - ' logger.go:42: 16:08:37 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.693209 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.693213 | controller | found' 2026-03-20 16:26:25.693218 | controller | - ' logger.go:42: 16:08:38 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.693222 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.693226 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.693230 | controller | - ' ' 2026-03-20 16:26:25.693234 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.693239 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.693243 | controller | - ' ' 2026-03-20 16:26:25.693247 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.693251 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.693255 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.693259 | controller | - ' exit 1' 2026-03-20 16:26:25.693264 | controller | - ' fi' 2026-03-20 16:26:25.693268 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.693272 | controller | - ' ' 2026-03-20 16:26:25.693278 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.693283 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.693289 | controller | | base64 -d)' 2026-03-20 16:26:25.693294 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.693298 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.693302 | controller | - ' exit 1' 2026-03-20 16:26:25.693306 | controller | - ' fi' 2026-03-20 16:26:25.693311 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.693316 | controller | - ' ' 2026-03-20 16:26:25.693321 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.693325 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.693329 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.693333 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.693338 | controller | --timeout=300s' 2026-03-20 16:26:25.693342 | controller | - ' ' 2026-03-20 16:26:25.693348 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.693352 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.693357 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.693361 | controller | - ' ' 2026-03-20 16:26:25.693365 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.693369 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.693374 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.693378 | controller | - ' ' 2026-03-20 16:26:25.693389 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.693394 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.693398 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.693402 | controller | available"' 2026-03-20 16:26:25.693406 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.693410 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.693415 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.693419 | controller | - ' exit 1' 2026-03-20 16:26:25.693423 | controller | - ' fi' 2026-03-20 16:26:25.693427 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.693431 | controller | - ' ' 2026-03-20 16:26:25.693436 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.693440 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.693444 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.693448 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.693452 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.693456 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.693461 | controller | -- \' 2026-03-20 16:26:25.693465 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.693469 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.693479 | controller | - ' logger.go:42: 16:08:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.693486 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.693490 | controller | - ' logger.go:42: 16:08:38 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.693495 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.693499 | controller | found' 2026-03-20 16:26:25.693503 | controller | - ' logger.go:42: 16:08:39 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.693507 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.693511 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.693516 | controller | - ' ' 2026-03-20 16:26:25.693520 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.693524 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.693528 | controller | - ' ' 2026-03-20 16:26:25.693532 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.693538 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.693542 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.693547 | controller | - ' exit 1' 2026-03-20 16:26:25.693551 | controller | - ' fi' 2026-03-20 16:26:25.693555 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.693559 | controller | - ' ' 2026-03-20 16:26:25.693563 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.693568 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.693572 | controller | | base64 -d)' 2026-03-20 16:26:25.693576 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.693580 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.693584 | controller | - ' exit 1' 2026-03-20 16:26:25.693589 | controller | - ' fi' 2026-03-20 16:26:25.693593 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.693598 | controller | - ' ' 2026-03-20 16:26:25.693602 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.693607 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.693611 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.693615 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.693619 | controller | --timeout=300s' 2026-03-20 16:26:25.693624 | controller | - ' ' 2026-03-20 16:26:25.693628 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.693632 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.693636 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.693640 | controller | - ' ' 2026-03-20 16:26:25.693645 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.693649 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.693653 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.693657 | controller | - ' ' 2026-03-20 16:26:25.693662 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.693673 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.693677 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.693681 | controller | available"' 2026-03-20 16:26:25.693689 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.693705 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.693711 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.693715 | controller | - ' exit 1' 2026-03-20 16:26:25.693723 | controller | - ' fi' 2026-03-20 16:26:25.693727 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.693732 | controller | - ' ' 2026-03-20 16:26:25.693736 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.693740 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.693744 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.693748 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.693753 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.693757 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.693761 | controller | -- \' 2026-03-20 16:26:25.693765 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.693769 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.693774 | controller | - ' logger.go:42: 16:08:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.693778 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.693782 | controller | - ' logger.go:42: 16:08:39 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.693786 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.693790 | controller | found' 2026-03-20 16:26:25.693795 | controller | - ' logger.go:42: 16:08:40 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.693799 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.693803 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.693808 | controller | - ' ' 2026-03-20 16:26:25.693812 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.693816 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.693820 | controller | - ' ' 2026-03-20 16:26:25.693825 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.693829 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.693833 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.693838 | controller | - ' exit 1' 2026-03-20 16:26:25.693842 | controller | - ' fi' 2026-03-20 16:26:25.693848 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.693853 | controller | - ' ' 2026-03-20 16:26:25.693857 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.693861 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.693865 | controller | | base64 -d)' 2026-03-20 16:26:25.693870 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.693874 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.693878 | controller | - ' exit 1' 2026-03-20 16:26:25.693882 | controller | - ' fi' 2026-03-20 16:26:25.693886 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.693891 | controller | - ' ' 2026-03-20 16:26:25.693895 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.693899 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.693906 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.693911 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.693915 | controller | --timeout=300s' 2026-03-20 16:26:25.693919 | controller | - ' ' 2026-03-20 16:26:25.693923 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.693928 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.693932 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.693936 | controller | - ' ' 2026-03-20 16:26:25.693940 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.693944 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.693949 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.693953 | controller | - ' ' 2026-03-20 16:26:25.693957 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.693961 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.693973 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.693978 | controller | available"' 2026-03-20 16:26:25.693983 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.693987 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.693991 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.693995 | controller | - ' exit 1' 2026-03-20 16:26:25.694000 | controller | - ' fi' 2026-03-20 16:26:25.694004 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.694008 | controller | - ' ' 2026-03-20 16:26:25.694013 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.694017 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.694021 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.694025 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.694029 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.694034 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.694038 | controller | -- \' 2026-03-20 16:26:25.694042 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.694046 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.694051 | controller | - ' logger.go:42: 16:08:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.694055 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.694059 | controller | - ' logger.go:42: 16:08:40 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.694064 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.694068 | controller | found' 2026-03-20 16:26:25.694072 | controller | - ' logger.go:42: 16:08:41 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.694076 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.694081 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.694087 | controller | - ' ' 2026-03-20 16:26:25.694091 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.694095 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.694099 | controller | - ' ' 2026-03-20 16:26:25.694104 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.694108 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.694112 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.694116 | controller | - ' exit 1' 2026-03-20 16:26:25.694121 | controller | - ' fi' 2026-03-20 16:26:25.694125 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.694132 | controller | - ' ' 2026-03-20 16:26:25.694136 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.694140 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.694145 | controller | | base64 -d)' 2026-03-20 16:26:25.694149 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.694153 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.694157 | controller | - ' exit 1' 2026-03-20 16:26:25.694162 | controller | - ' fi' 2026-03-20 16:26:25.694166 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.694170 | controller | - ' ' 2026-03-20 16:26:25.694174 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.694179 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.694183 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.694187 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.694191 | controller | --timeout=300s' 2026-03-20 16:26:25.694195 | controller | - ' ' 2026-03-20 16:26:25.694200 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.694204 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.694208 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.694212 | controller | - ' ' 2026-03-20 16:26:25.694217 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.694221 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.694225 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.694229 | controller | - ' ' 2026-03-20 16:26:25.694234 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.694238 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.694249 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.694255 | controller | available"' 2026-03-20 16:26:25.694259 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.694263 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.694267 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.694271 | controller | - ' exit 1' 2026-03-20 16:26:25.694276 | controller | - ' fi' 2026-03-20 16:26:25.694280 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.694284 | controller | - ' ' 2026-03-20 16:26:25.694291 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.694295 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.694300 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.694305 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.694310 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.694314 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.694319 | controller | -- \' 2026-03-20 16:26:25.694324 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.694329 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.694333 | controller | - ' logger.go:42: 16:08:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.694337 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.694342 | controller | - ' logger.go:42: 16:08:41 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.694346 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.694350 | controller | found' 2026-03-20 16:26:25.694354 | controller | - ' logger.go:42: 16:08:42 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.694358 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.694362 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.694367 | controller | - ' ' 2026-03-20 16:26:25.694371 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.694375 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.694379 | controller | - ' ' 2026-03-20 16:26:25.694383 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.694388 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.694392 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.694396 | controller | - ' exit 1' 2026-03-20 16:26:25.694400 | controller | - ' fi' 2026-03-20 16:26:25.694404 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.694409 | controller | - ' ' 2026-03-20 16:26:25.694415 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.694419 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.694423 | controller | | base64 -d)' 2026-03-20 16:26:25.694428 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.694432 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.694436 | controller | - ' exit 1' 2026-03-20 16:26:25.694440 | controller | - ' fi' 2026-03-20 16:26:25.694444 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.694449 | controller | - ' ' 2026-03-20 16:26:25.694453 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.694457 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.694461 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.694465 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.694470 | controller | --timeout=300s' 2026-03-20 16:26:25.694474 | controller | - ' ' 2026-03-20 16:26:25.694478 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.694482 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.694488 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.694492 | controller | - ' ' 2026-03-20 16:26:25.694497 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.694501 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.694505 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.694509 | controller | - ' ' 2026-03-20 16:26:25.694513 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.694519 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.694523 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.694528 | controller | available"' 2026-03-20 16:26:25.694539 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.694544 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.694548 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.694553 | controller | - ' exit 1' 2026-03-20 16:26:25.694557 | controller | - ' fi' 2026-03-20 16:26:25.694561 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.694567 | controller | - ' ' 2026-03-20 16:26:25.694571 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.694576 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.694580 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.694584 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.694588 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.694592 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.694596 | controller | -- \' 2026-03-20 16:26:25.694601 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.694605 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.694609 | controller | - ' logger.go:42: 16:08:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.694613 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.694617 | controller | - ' logger.go:42: 16:08:42 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.694622 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.694626 | controller | found' 2026-03-20 16:26:25.694630 | controller | - ' logger.go:42: 16:08:43 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.694634 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.694638 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.694643 | controller | - ' ' 2026-03-20 16:26:25.694647 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.694651 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.694655 | controller | - ' ' 2026-03-20 16:26:25.694659 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.694663 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.694668 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.694676 | controller | - ' exit 1' 2026-03-20 16:26:25.694680 | controller | - ' fi' 2026-03-20 16:26:25.694685 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.694689 | controller | - ' ' 2026-03-20 16:26:25.694710 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.694715 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.694719 | controller | | base64 -d)' 2026-03-20 16:26:25.694723 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.694727 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.694732 | controller | - ' exit 1' 2026-03-20 16:26:25.694736 | controller | - ' fi' 2026-03-20 16:26:25.694740 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.694744 | controller | - ' ' 2026-03-20 16:26:25.694748 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.694753 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.694759 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.694763 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.694768 | controller | --timeout=300s' 2026-03-20 16:26:25.694772 | controller | - ' ' 2026-03-20 16:26:25.694776 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.694780 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.694785 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.694789 | controller | - ' ' 2026-03-20 16:26:25.694793 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.694797 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.694801 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.694806 | controller | - ' ' 2026-03-20 16:26:25.694810 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.694814 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.694818 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.694822 | controller | available"' 2026-03-20 16:26:25.694827 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.694839 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.694843 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.694847 | controller | - ' exit 1' 2026-03-20 16:26:25.694852 | controller | - ' fi' 2026-03-20 16:26:25.694856 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.694860 | controller | - ' ' 2026-03-20 16:26:25.694865 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.694869 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.694873 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.694877 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.694882 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.694888 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.694892 | controller | -- \' 2026-03-20 16:26:25.694896 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.694901 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.694905 | controller | - ' logger.go:42: 16:08:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.694909 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.694913 | controller | - ' logger.go:42: 16:08:43 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.694917 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.694922 | controller | found' 2026-03-20 16:26:25.694926 | controller | - ' logger.go:42: 16:08:44 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.694930 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.694934 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.694938 | controller | - ' ' 2026-03-20 16:26:25.694944 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.694948 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.694952 | controller | - ' ' 2026-03-20 16:26:25.694956 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.694960 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.694965 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.694969 | controller | - ' exit 1' 2026-03-20 16:26:25.694973 | controller | - ' fi' 2026-03-20 16:26:25.694977 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.694981 | controller | - ' ' 2026-03-20 16:26:25.694986 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.694990 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.694994 | controller | | base64 -d)' 2026-03-20 16:26:25.694999 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.695003 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.695007 | controller | - ' exit 1' 2026-03-20 16:26:25.695011 | controller | - ' fi' 2026-03-20 16:26:25.695015 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.695020 | controller | - ' ' 2026-03-20 16:26:25.695024 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.695028 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.695032 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.695036 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.695040 | controller | --timeout=300s' 2026-03-20 16:26:25.695045 | controller | - ' ' 2026-03-20 16:26:25.695049 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.695053 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.695057 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.695061 | controller | - ' ' 2026-03-20 16:26:25.695066 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.695070 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.695074 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.695080 | controller | - ' ' 2026-03-20 16:26:25.695084 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.695088 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.695093 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.695097 | controller | available"' 2026-03-20 16:26:25.695101 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.695112 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.695117 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.695122 | controller | - ' exit 1' 2026-03-20 16:26:25.695126 | controller | - ' fi' 2026-03-20 16:26:25.695130 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.695135 | controller | - ' ' 2026-03-20 16:26:25.695139 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.695143 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.695147 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.695151 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.695156 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.695160 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.695164 | controller | -- \' 2026-03-20 16:26:25.695168 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.695172 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.695177 | controller | - ' logger.go:42: 16:08:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.695181 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.695185 | controller | - ' logger.go:42: 16:08:45 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.695189 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.695193 | controller | found' 2026-03-20 16:26:25.695198 | controller | - ' logger.go:42: 16:08:46 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.695202 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.695206 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.695210 | controller | - ' ' 2026-03-20 16:26:25.695215 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.695219 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.695223 | controller | - ' ' 2026-03-20 16:26:25.695227 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.695231 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.695235 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.695240 | controller | - ' exit 1' 2026-03-20 16:26:25.695244 | controller | - ' fi' 2026-03-20 16:26:25.695248 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.695252 | controller | - ' ' 2026-03-20 16:26:25.695256 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.695261 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.695265 | controller | | base64 -d)' 2026-03-20 16:26:25.695269 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.695277 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.695282 | controller | - ' exit 1' 2026-03-20 16:26:25.695286 | controller | - ' fi' 2026-03-20 16:26:25.695290 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.695294 | controller | - ' ' 2026-03-20 16:26:25.695299 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.695303 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.695307 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.695313 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.695318 | controller | --timeout=300s' 2026-03-20 16:26:25.695322 | controller | - ' ' 2026-03-20 16:26:25.695326 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.695330 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.695335 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.695339 | controller | - ' ' 2026-03-20 16:26:25.695343 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.695347 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.695351 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.695356 | controller | - ' ' 2026-03-20 16:26:25.695360 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.695364 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.695368 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.695372 | controller | available"' 2026-03-20 16:26:25.695377 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.695381 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.695392 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.695397 | controller | - ' exit 1' 2026-03-20 16:26:25.695401 | controller | - ' fi' 2026-03-20 16:26:25.695406 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.695410 | controller | - ' ' 2026-03-20 16:26:25.695416 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.695421 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.695426 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.695430 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.695434 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.695438 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.695442 | controller | -- \' 2026-03-20 16:26:25.695447 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.695451 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.695455 | controller | - ' logger.go:42: 16:08:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.695461 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.695465 | controller | - ' logger.go:42: 16:08:46 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.695470 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.695474 | controller | found' 2026-03-20 16:26:25.695478 | controller | - ' logger.go:42: 16:08:47 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.695482 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.695487 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.695491 | controller | - ' ' 2026-03-20 16:26:25.695495 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.695499 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.695504 | controller | - ' ' 2026-03-20 16:26:25.695508 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.695512 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.695516 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.695520 | controller | - ' exit 1' 2026-03-20 16:26:25.695525 | controller | - ' fi' 2026-03-20 16:26:25.695529 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.695533 | controller | - ' ' 2026-03-20 16:26:25.695537 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.695541 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.695546 | controller | | base64 -d)' 2026-03-20 16:26:25.695550 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.695554 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.695558 | controller | - ' exit 1' 2026-03-20 16:26:25.695562 | controller | - ' fi' 2026-03-20 16:26:25.695567 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.695571 | controller | - ' ' 2026-03-20 16:26:25.695575 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.695579 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.695583 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.695588 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.695610 | controller | --timeout=300s' 2026-03-20 16:26:25.695616 | controller | - ' ' 2026-03-20 16:26:25.695621 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.695627 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.695634 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.695639 | controller | - ' ' 2026-03-20 16:26:25.695644 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.695649 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.695654 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.695659 | controller | - ' ' 2026-03-20 16:26:25.695664 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.695669 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.695673 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.695678 | controller | available"' 2026-03-20 16:26:25.695683 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.695690 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.695709 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.695723 | controller | - ' exit 1' 2026-03-20 16:26:25.695728 | controller | - ' fi' 2026-03-20 16:26:25.695733 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.695738 | controller | - ' ' 2026-03-20 16:26:25.695743 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.695748 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.695753 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.695758 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.695763 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.695768 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.695772 | controller | -- \' 2026-03-20 16:26:25.695777 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.695782 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.695787 | controller | - ' logger.go:42: 16:08:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.695792 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.695797 | controller | - ' logger.go:42: 16:08:47 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.695802 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.695807 | controller | found' 2026-03-20 16:26:25.695812 | controller | - ' logger.go:42: 16:08:48 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.695817 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.695821 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.695826 | controller | - ' ' 2026-03-20 16:26:25.695831 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.695836 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.695841 | controller | - ' ' 2026-03-20 16:26:25.695846 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.695851 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.695855 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.695860 | controller | - ' exit 1' 2026-03-20 16:26:25.695864 | controller | - ' fi' 2026-03-20 16:26:25.695868 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.695872 | controller | - ' ' 2026-03-20 16:26:25.695879 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.695883 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.695887 | controller | | base64 -d)' 2026-03-20 16:26:25.695891 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.695895 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.695900 | controller | - ' exit 1' 2026-03-20 16:26:25.695904 | controller | - ' fi' 2026-03-20 16:26:25.695908 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.695914 | controller | - ' ' 2026-03-20 16:26:25.695918 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.695922 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.695937 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.695941 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.695946 | controller | --timeout=300s' 2026-03-20 16:26:25.695950 | controller | - ' ' 2026-03-20 16:26:25.695954 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.695958 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.695963 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.695967 | controller | - ' ' 2026-03-20 16:26:25.695971 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.695975 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.695979 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.695984 | controller | - ' ' 2026-03-20 16:26:25.695996 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.696000 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.696006 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.696010 | controller | available"' 2026-03-20 16:26:25.696014 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.696019 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.696023 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.696027 | controller | - ' exit 1' 2026-03-20 16:26:25.696031 | controller | - ' fi' 2026-03-20 16:26:25.696043 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.696048 | controller | - ' ' 2026-03-20 16:26:25.696052 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.696056 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.696060 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.696064 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.696069 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.696073 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.696077 | controller | -- \' 2026-03-20 16:26:25.696081 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.696086 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.696090 | controller | - ' logger.go:42: 16:08:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.696094 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.696098 | controller | - ' logger.go:42: 16:08:48 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.696102 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.696107 | controller | found' 2026-03-20 16:26:25.696111 | controller | - ' logger.go:42: 16:08:49 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.696115 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.696119 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.696124 | controller | - ' ' 2026-03-20 16:26:25.696130 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.696134 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.696138 | controller | - ' ' 2026-03-20 16:26:25.696142 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.696146 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.696151 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.696155 | controller | - ' exit 1' 2026-03-20 16:26:25.696159 | controller | - ' fi' 2026-03-20 16:26:25.696163 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.696168 | controller | - ' ' 2026-03-20 16:26:25.696172 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.696176 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.696180 | controller | | base64 -d)' 2026-03-20 16:26:25.696184 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.696189 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.696193 | controller | - ' exit 1' 2026-03-20 16:26:25.696197 | controller | - ' fi' 2026-03-20 16:26:25.696201 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.696205 | controller | - ' ' 2026-03-20 16:26:25.696210 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.696214 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.696218 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.696222 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.696226 | controller | --timeout=300s' 2026-03-20 16:26:25.696230 | controller | - ' ' 2026-03-20 16:26:25.696235 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.696239 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.696243 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.696247 | controller | - ' ' 2026-03-20 16:26:25.696252 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.696256 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.696260 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.696264 | controller | - ' ' 2026-03-20 16:26:25.696268 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.696273 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.696277 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.696281 | controller | available"' 2026-03-20 16:26:25.696285 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.696290 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.696294 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.696298 | controller | - ' exit 1' 2026-03-20 16:26:25.696302 | controller | - ' fi' 2026-03-20 16:26:25.696314 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.696319 | controller | - ' ' 2026-03-20 16:26:25.696323 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.696329 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.696333 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.696338 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.696342 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.696346 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.696350 | controller | -- \' 2026-03-20 16:26:25.696354 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.696358 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.696363 | controller | - ' logger.go:42: 16:08:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.696367 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.696371 | controller | - ' logger.go:42: 16:08:49 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.696375 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.696379 | controller | found' 2026-03-20 16:26:25.696384 | controller | - ' logger.go:42: 16:08:50 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.696388 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.696392 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.696396 | controller | - ' ' 2026-03-20 16:26:25.696400 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.696405 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.696409 | controller | - ' ' 2026-03-20 16:26:25.696413 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.696418 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.696422 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.696427 | controller | - ' exit 1' 2026-03-20 16:26:25.696431 | controller | - ' fi' 2026-03-20 16:26:25.696435 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.696439 | controller | - ' ' 2026-03-20 16:26:25.696444 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.696448 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.696452 | controller | | base64 -d)' 2026-03-20 16:26:25.696456 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.696460 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.696464 | controller | - ' exit 1' 2026-03-20 16:26:25.696469 | controller | - ' fi' 2026-03-20 16:26:25.696473 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.696477 | controller | - ' ' 2026-03-20 16:26:25.696481 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.696485 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.696490 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.696494 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.696498 | controller | --timeout=300s' 2026-03-20 16:26:25.696502 | controller | - ' ' 2026-03-20 16:26:25.696506 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.696510 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.696516 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.696521 | controller | - ' ' 2026-03-20 16:26:25.696527 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.696531 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.696535 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.696540 | controller | - ' ' 2026-03-20 16:26:25.696544 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.696548 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.696556 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.696560 | controller | available"' 2026-03-20 16:26:25.696564 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.696569 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.696573 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.696577 | controller | - ' exit 1' 2026-03-20 16:26:25.696581 | controller | - ' fi' 2026-03-20 16:26:25.696585 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.696590 | controller | - ' ' 2026-03-20 16:26:25.696601 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.696606 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.696610 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.696614 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.696621 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.696625 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.696629 | controller | -- \' 2026-03-20 16:26:25.696633 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.696637 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.696642 | controller | - ' logger.go:42: 16:08:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.696646 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.696650 | controller | - ' logger.go:42: 16:08:50 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.696654 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.696658 | controller | found' 2026-03-20 16:26:25.696662 | controller | - ' logger.go:42: 16:08:51 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.696667 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.696671 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.696675 | controller | - ' ' 2026-03-20 16:26:25.696679 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.696683 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.696688 | controller | - ' ' 2026-03-20 16:26:25.696703 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.696709 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.696714 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.696718 | controller | - ' exit 1' 2026-03-20 16:26:25.696724 | controller | - ' fi' 2026-03-20 16:26:25.696728 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.696732 | controller | - ' ' 2026-03-20 16:26:25.696737 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.696741 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.696745 | controller | | base64 -d)' 2026-03-20 16:26:25.696749 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.696753 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.696758 | controller | - ' exit 1' 2026-03-20 16:26:25.696762 | controller | - ' fi' 2026-03-20 16:26:25.696767 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.696771 | controller | - ' ' 2026-03-20 16:26:25.696776 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.696780 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.696784 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.696788 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.696792 | controller | --timeout=300s' 2026-03-20 16:26:25.696797 | controller | - ' ' 2026-03-20 16:26:25.696801 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.696805 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.696809 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.696813 | controller | - ' ' 2026-03-20 16:26:25.696818 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.696822 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.696826 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.696830 | controller | - ' ' 2026-03-20 16:26:25.696834 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.696839 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.696843 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.696847 | controller | available"' 2026-03-20 16:26:25.696851 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.696856 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.696860 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.696864 | controller | - ' exit 1' 2026-03-20 16:26:25.696868 | controller | - ' fi' 2026-03-20 16:26:25.696872 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.696877 | controller | - ' ' 2026-03-20 16:26:25.696888 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.696893 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.696898 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.696902 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.696906 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.696910 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.696916 | controller | -- \' 2026-03-20 16:26:25.696921 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.696925 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.696929 | controller | - ' logger.go:42: 16:08:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.696933 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.696937 | controller | - ' logger.go:42: 16:08:52 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.696941 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.696946 | controller | found' 2026-03-20 16:26:25.696950 | controller | - ' logger.go:42: 16:08:53 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.696954 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.696958 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.696962 | controller | - ' ' 2026-03-20 16:26:25.696967 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.696971 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.696975 | controller | - ' ' 2026-03-20 16:26:25.696979 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.696983 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.696987 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.696992 | controller | - ' exit 1' 2026-03-20 16:26:25.696996 | controller | - ' fi' 2026-03-20 16:26:25.697000 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.697004 | controller | - ' ' 2026-03-20 16:26:25.697008 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.697013 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.697017 | controller | | base64 -d)' 2026-03-20 16:26:25.697021 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.697025 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.697029 | controller | - ' exit 1' 2026-03-20 16:26:25.697034 | controller | - ' fi' 2026-03-20 16:26:25.697038 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.697042 | controller | - ' ' 2026-03-20 16:26:25.697046 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.697050 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.697054 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.697058 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.697063 | controller | --timeout=300s' 2026-03-20 16:26:25.697067 | controller | - ' ' 2026-03-20 16:26:25.697071 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.697075 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.697079 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.697084 | controller | - ' ' 2026-03-20 16:26:25.697088 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.697092 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.697096 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.697102 | controller | - ' ' 2026-03-20 16:26:25.697106 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.697110 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.697115 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.697119 | controller | available"' 2026-03-20 16:26:25.697123 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.697130 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.697134 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.697139 | controller | - ' exit 1' 2026-03-20 16:26:25.697143 | controller | - ' fi' 2026-03-20 16:26:25.697149 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.697153 | controller | - ' ' 2026-03-20 16:26:25.697157 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.697169 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.697174 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.697178 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.697182 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.697186 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.697191 | controller | -- \' 2026-03-20 16:26:25.697195 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.697199 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.697204 | controller | - ' logger.go:42: 16:08:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.697208 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.697212 | controller | - ' logger.go:42: 16:08:53 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.697216 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.697220 | controller | found' 2026-03-20 16:26:25.697225 | controller | - ' logger.go:42: 16:08:54 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.697229 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.697233 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.697237 | controller | - ' ' 2026-03-20 16:26:25.697241 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.697246 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.697250 | controller | - ' ' 2026-03-20 16:26:25.697254 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.697258 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.697262 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.697267 | controller | - ' exit 1' 2026-03-20 16:26:25.697271 | controller | - ' fi' 2026-03-20 16:26:25.697275 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.697279 | controller | - ' ' 2026-03-20 16:26:25.697283 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.697288 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.697292 | controller | | base64 -d)' 2026-03-20 16:26:25.697296 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.697300 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.697314 | controller | - ' exit 1' 2026-03-20 16:26:25.697319 | controller | - ' fi' 2026-03-20 16:26:25.697323 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.697329 | controller | - ' ' 2026-03-20 16:26:25.697334 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.697338 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.697342 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.697347 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.697351 | controller | --timeout=300s' 2026-03-20 16:26:25.697355 | controller | - ' ' 2026-03-20 16:26:25.697359 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.697363 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.697368 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.697372 | controller | - ' ' 2026-03-20 16:26:25.697376 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.697380 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.697384 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.697389 | controller | - ' ' 2026-03-20 16:26:25.697393 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.697397 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.697401 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.697405 | controller | available"' 2026-03-20 16:26:25.697410 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.697414 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.697418 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.697422 | controller | - ' exit 1' 2026-03-20 16:26:25.697426 | controller | - ' fi' 2026-03-20 16:26:25.697431 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.697435 | controller | - ' ' 2026-03-20 16:26:25.697439 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.697443 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.697455 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.697460 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.697464 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.697469 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.697473 | controller | -- \' 2026-03-20 16:26:25.697477 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.697481 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.697485 | controller | - ' logger.go:42: 16:08:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.697490 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.697496 | controller | - ' logger.go:42: 16:08:54 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.697502 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.697506 | controller | found' 2026-03-20 16:26:25.697511 | controller | - ' logger.go:42: 16:08:55 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.697515 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.697519 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.697524 | controller | - ' ' 2026-03-20 16:26:25.697528 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.697532 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.697536 | controller | - ' ' 2026-03-20 16:26:25.697540 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.697544 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.697549 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.697553 | controller | - ' exit 1' 2026-03-20 16:26:25.697557 | controller | - ' fi' 2026-03-20 16:26:25.697561 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.697565 | controller | - ' ' 2026-03-20 16:26:25.697570 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.697574 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.697578 | controller | | base64 -d)' 2026-03-20 16:26:25.697582 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.697586 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.697591 | controller | - ' exit 1' 2026-03-20 16:26:25.697597 | controller | - ' fi' 2026-03-20 16:26:25.697601 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.697605 | controller | - ' ' 2026-03-20 16:26:25.697609 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.697613 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.697618 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.697622 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.697626 | controller | --timeout=300s' 2026-03-20 16:26:25.697632 | controller | - ' ' 2026-03-20 16:26:25.697636 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.697640 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.697644 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.697649 | controller | - ' ' 2026-03-20 16:26:25.697653 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.697657 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.697661 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.697665 | controller | - ' ' 2026-03-20 16:26:25.697670 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.697674 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.697678 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.697682 | controller | available"' 2026-03-20 16:26:25.697687 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.697691 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.697711 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.697715 | controller | - ' exit 1' 2026-03-20 16:26:25.697722 | controller | - ' fi' 2026-03-20 16:26:25.697726 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.697730 | controller | - ' ' 2026-03-20 16:26:25.697734 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.697739 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.697754 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.697759 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.697763 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.697767 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.697771 | controller | -- \' 2026-03-20 16:26:25.697775 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.697780 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.697784 | controller | - ' logger.go:42: 16:08:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.697788 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.697792 | controller | - ' logger.go:42: 16:08:55 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.697796 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.697801 | controller | found' 2026-03-20 16:26:25.697805 | controller | - ' logger.go:42: 16:08:56 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.697809 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.697813 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.697818 | controller | - ' ' 2026-03-20 16:26:25.697822 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.697826 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.697830 | controller | - ' ' 2026-03-20 16:26:25.697834 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.697838 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.697843 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.697847 | controller | - ' exit 1' 2026-03-20 16:26:25.697851 | controller | - ' fi' 2026-03-20 16:26:25.697856 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.697860 | controller | - ' ' 2026-03-20 16:26:25.697864 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.697868 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.697872 | controller | | base64 -d)' 2026-03-20 16:26:25.697877 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.697881 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.697885 | controller | - ' exit 1' 2026-03-20 16:26:25.697889 | controller | - ' fi' 2026-03-20 16:26:25.697895 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.697899 | controller | - ' ' 2026-03-20 16:26:25.697903 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.697907 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.697911 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.697917 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.697922 | controller | --timeout=300s' 2026-03-20 16:26:25.697926 | controller | - ' ' 2026-03-20 16:26:25.697930 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.697934 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.697938 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.697943 | controller | - ' ' 2026-03-20 16:26:25.697947 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.697951 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.697955 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.697959 | controller | - ' ' 2026-03-20 16:26:25.697963 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.697968 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.697972 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.697976 | controller | available"' 2026-03-20 16:26:25.697980 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.697985 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.697989 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.697993 | controller | - ' exit 1' 2026-03-20 16:26:25.697997 | controller | - ' fi' 2026-03-20 16:26:25.698001 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.698006 | controller | - ' ' 2026-03-20 16:26:25.698010 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.698014 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.698025 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.698030 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.698034 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.698038 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.698043 | controller | -- \' 2026-03-20 16:26:25.698047 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.698051 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.698055 | controller | - ' logger.go:42: 16:08:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.698059 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.698064 | controller | - ' logger.go:42: 16:08:56 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.698068 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.698072 | controller | found' 2026-03-20 16:26:25.698076 | controller | - ' logger.go:42: 16:08:57 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.698080 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.698085 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.698089 | controller | - ' ' 2026-03-20 16:26:25.698093 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.698099 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.698103 | controller | - ' ' 2026-03-20 16:26:25.698107 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.698112 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.698116 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.698120 | controller | - ' exit 1' 2026-03-20 16:26:25.698124 | controller | - ' fi' 2026-03-20 16:26:25.698128 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.698133 | controller | - ' ' 2026-03-20 16:26:25.698137 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.698141 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.698145 | controller | | base64 -d)' 2026-03-20 16:26:25.698149 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.698154 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.698158 | controller | - ' exit 1' 2026-03-20 16:26:25.698162 | controller | - ' fi' 2026-03-20 16:26:25.698166 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.698170 | controller | - ' ' 2026-03-20 16:26:25.698175 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.698179 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.698183 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.698187 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.698191 | controller | --timeout=300s' 2026-03-20 16:26:25.698196 | controller | - ' ' 2026-03-20 16:26:25.698200 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.698204 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.698208 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.698212 | controller | - ' ' 2026-03-20 16:26:25.698217 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.698221 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.698225 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.698229 | controller | - ' ' 2026-03-20 16:26:25.698234 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.698240 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.698244 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.698248 | controller | available"' 2026-03-20 16:26:25.698252 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.698257 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.698261 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.698265 | controller | - ' exit 1' 2026-03-20 16:26:25.698269 | controller | - ' fi' 2026-03-20 16:26:25.698273 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.698278 | controller | - ' ' 2026-03-20 16:26:25.698282 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.698286 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.698293 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.698306 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.698311 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.698315 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.698320 | controller | -- \' 2026-03-20 16:26:25.698324 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.698328 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.698332 | controller | - ' logger.go:42: 16:08:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.698337 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.698341 | controller | - ' logger.go:42: 16:08:57 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.698345 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.698349 | controller | found' 2026-03-20 16:26:25.698353 | controller | - ' logger.go:42: 16:08:58 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.698357 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.698362 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.698366 | controller | - ' ' 2026-03-20 16:26:25.698370 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.698374 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.698378 | controller | - ' ' 2026-03-20 16:26:25.698382 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.698387 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.698391 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.698395 | controller | - ' exit 1' 2026-03-20 16:26:25.698399 | controller | - ' fi' 2026-03-20 16:26:25.698403 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.698408 | controller | - ' ' 2026-03-20 16:26:25.698414 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.698418 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.698422 | controller | | base64 -d)' 2026-03-20 16:26:25.698426 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.698431 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.698435 | controller | - ' exit 1' 2026-03-20 16:26:25.698439 | controller | - ' fi' 2026-03-20 16:26:25.698443 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.698447 | controller | - ' ' 2026-03-20 16:26:25.698452 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.698456 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.698460 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.698464 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.698468 | controller | --timeout=300s' 2026-03-20 16:26:25.698472 | controller | - ' ' 2026-03-20 16:26:25.698477 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.698481 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.698485 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.698491 | controller | - ' ' 2026-03-20 16:26:25.698495 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.698499 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.698504 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.698508 | controller | - ' ' 2026-03-20 16:26:25.698512 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.698516 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.698520 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.698525 | controller | available"' 2026-03-20 16:26:25.698529 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.698533 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.698537 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.698541 | controller | - ' exit 1' 2026-03-20 16:26:25.698546 | controller | - ' fi' 2026-03-20 16:26:25.698550 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.698554 | controller | - ' ' 2026-03-20 16:26:25.698558 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.698562 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.698567 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.698571 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.698583 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.698587 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.698593 | controller | -- \' 2026-03-20 16:26:25.698598 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.698602 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.698606 | controller | - ' logger.go:42: 16:08:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.698610 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.698615 | controller | - ' logger.go:42: 16:08:59 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.698619 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.698623 | controller | found' 2026-03-20 16:26:25.698627 | controller | - ' logger.go:42: 16:09:00 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.698632 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.698636 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.698640 | controller | - ' ' 2026-03-20 16:26:25.698644 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.698649 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.698653 | controller | - ' ' 2026-03-20 16:26:25.698659 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.698663 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.698667 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.698673 | controller | - ' exit 1' 2026-03-20 16:26:25.698677 | controller | - ' fi' 2026-03-20 16:26:25.698683 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.698687 | controller | - ' ' 2026-03-20 16:26:25.698702 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.698708 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.698712 | controller | | base64 -d)' 2026-03-20 16:26:25.698717 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.698721 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.698725 | controller | - ' exit 1' 2026-03-20 16:26:25.698729 | controller | - ' fi' 2026-03-20 16:26:25.698734 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.698738 | controller | - ' ' 2026-03-20 16:26:25.698742 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.698746 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.698750 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.698754 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.698758 | controller | --timeout=300s' 2026-03-20 16:26:25.698763 | controller | - ' ' 2026-03-20 16:26:25.698767 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.698771 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.698775 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.698779 | controller | - ' ' 2026-03-20 16:26:25.698784 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.698788 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.698792 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.698796 | controller | - ' ' 2026-03-20 16:26:25.698800 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.698804 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.698809 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.698813 | controller | available"' 2026-03-20 16:26:25.698817 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.698821 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.698825 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.698830 | controller | - ' exit 1' 2026-03-20 16:26:25.698834 | controller | - ' fi' 2026-03-20 16:26:25.698838 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.698842 | controller | - ' ' 2026-03-20 16:26:25.698846 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.698851 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.698855 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.698859 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.698873 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.698878 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.698884 | controller | -- \' 2026-03-20 16:26:25.698889 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.698893 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.698897 | controller | - ' logger.go:42: 16:09:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.698901 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.698906 | controller | - ' logger.go:42: 16:09:00 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.698910 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.698914 | controller | found' 2026-03-20 16:26:25.698918 | controller | - ' logger.go:42: 16:09:01 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.698922 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.698927 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.698931 | controller | - ' ' 2026-03-20 16:26:25.698935 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.698939 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.698943 | controller | - ' ' 2026-03-20 16:26:25.698947 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.698952 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.698956 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.698960 | controller | - ' exit 1' 2026-03-20 16:26:25.698964 | controller | - ' fi' 2026-03-20 16:26:25.698968 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.698972 | controller | - ' ' 2026-03-20 16:26:25.698977 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.698981 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.698985 | controller | | base64 -d)' 2026-03-20 16:26:25.698989 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.698993 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.698998 | controller | - ' exit 1' 2026-03-20 16:26:25.699002 | controller | - ' fi' 2026-03-20 16:26:25.699006 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.699010 | controller | - ' ' 2026-03-20 16:26:25.699014 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.699019 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.699023 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.699027 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.699031 | controller | --timeout=300s' 2026-03-20 16:26:25.699035 | controller | - ' ' 2026-03-20 16:26:25.699040 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.699044 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.699048 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.699052 | controller | - ' ' 2026-03-20 16:26:25.699056 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.699061 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.699065 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.699069 | controller | - ' ' 2026-03-20 16:26:25.699075 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.699079 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.699083 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.699088 | controller | available"' 2026-03-20 16:26:25.699092 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.699096 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.699100 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.699104 | controller | - ' exit 1' 2026-03-20 16:26:25.699108 | controller | - ' fi' 2026-03-20 16:26:25.699113 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.699117 | controller | - ' ' 2026-03-20 16:26:25.699121 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.699125 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.699129 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.699133 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.699146 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.699151 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.699155 | controller | -- \' 2026-03-20 16:26:25.699159 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.699163 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.699168 | controller | - ' logger.go:42: 16:09:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.699172 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.699176 | controller | - ' logger.go:42: 16:09:01 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.699180 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.699185 | controller | found' 2026-03-20 16:26:25.699189 | controller | - ' logger.go:42: 16:09:02 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.699193 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.699197 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.699201 | controller | - ' ' 2026-03-20 16:26:25.699205 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.699210 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.699214 | controller | - ' ' 2026-03-20 16:26:25.699218 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.699222 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.699227 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.699231 | controller | - ' exit 1' 2026-03-20 16:26:25.699235 | controller | - ' fi' 2026-03-20 16:26:25.699239 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.699243 | controller | - ' ' 2026-03-20 16:26:25.699247 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.699252 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.699256 | controller | | base64 -d)' 2026-03-20 16:26:25.699260 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.699264 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.699272 | controller | - ' exit 1' 2026-03-20 16:26:25.699276 | controller | - ' fi' 2026-03-20 16:26:25.699280 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.699285 | controller | - ' ' 2026-03-20 16:26:25.699289 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.699293 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.699297 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.699301 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.699305 | controller | --timeout=300s' 2026-03-20 16:26:25.699310 | controller | - ' ' 2026-03-20 16:26:25.699314 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.699318 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.699322 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.699326 | controller | - ' ' 2026-03-20 16:26:25.699331 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.699335 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.699339 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.699343 | controller | - ' ' 2026-03-20 16:26:25.699347 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.699352 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.699356 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.699360 | controller | available"' 2026-03-20 16:26:25.699364 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.699368 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.699373 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.699377 | controller | - ' exit 1' 2026-03-20 16:26:25.699381 | controller | - ' fi' 2026-03-20 16:26:25.699385 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.699389 | controller | - ' ' 2026-03-20 16:26:25.699394 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.699398 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.699402 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.699406 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.699410 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.699421 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.699426 | controller | -- \' 2026-03-20 16:26:25.699430 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.699434 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.699438 | controller | - ' logger.go:42: 16:09:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.699442 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.699460 | controller | - ' logger.go:42: 16:09:02 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.699467 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.699471 | controller | found' 2026-03-20 16:26:25.699475 | controller | - ' logger.go:42: 16:09:03 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.699479 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.699484 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.699488 | controller | - ' ' 2026-03-20 16:26:25.699492 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.699496 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.699500 | controller | - ' ' 2026-03-20 16:26:25.699505 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.699509 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.699513 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.699517 | controller | - ' exit 1' 2026-03-20 16:26:25.699521 | controller | - ' fi' 2026-03-20 16:26:25.699525 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.699530 | controller | - ' ' 2026-03-20 16:26:25.699534 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.699538 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.699542 | controller | | base64 -d)' 2026-03-20 16:26:25.699546 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.699551 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.699555 | controller | - ' exit 1' 2026-03-20 16:26:25.699567 | controller | - ' fi' 2026-03-20 16:26:25.699572 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.699576 | controller | - ' ' 2026-03-20 16:26:25.699580 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.699584 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.699589 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.699593 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.699612 | controller | --timeout=300s' 2026-03-20 16:26:25.699618 | controller | - ' ' 2026-03-20 16:26:25.699623 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.699628 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.699632 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.699637 | controller | - ' ' 2026-03-20 16:26:25.699642 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.699647 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.699655 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.699660 | controller | - ' ' 2026-03-20 16:26:25.699665 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.699670 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.699675 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.699680 | controller | available"' 2026-03-20 16:26:25.699686 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.699712 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.699721 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.699726 | controller | - ' exit 1' 2026-03-20 16:26:25.699731 | controller | - ' fi' 2026-03-20 16:26:25.699736 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.699741 | controller | - ' ' 2026-03-20 16:26:25.699746 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.699751 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.699756 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.699761 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.699766 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.699770 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.699786 | controller | -- \' 2026-03-20 16:26:25.699791 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.699796 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.699801 | controller | - ' logger.go:42: 16:09:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.699806 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.699811 | controller | - ' logger.go:42: 16:09:03 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.699816 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.699821 | controller | found' 2026-03-20 16:26:25.699826 | controller | - ' logger.go:42: 16:09:04 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.699831 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.699835 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.699840 | controller | - ' ' 2026-03-20 16:26:25.699845 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.699850 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.699863 | controller | - ' ' 2026-03-20 16:26:25.699868 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.699872 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.699876 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.699880 | controller | - ' exit 1' 2026-03-20 16:26:25.699885 | controller | - ' fi' 2026-03-20 16:26:25.699889 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.699893 | controller | - ' ' 2026-03-20 16:26:25.699897 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.699901 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.699905 | controller | | base64 -d)' 2026-03-20 16:26:25.699910 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.699914 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.699918 | controller | - ' exit 1' 2026-03-20 16:26:25.699922 | controller | - ' fi' 2026-03-20 16:26:25.699927 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.699931 | controller | - ' ' 2026-03-20 16:26:25.699935 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.699939 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.699943 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.699949 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.699954 | controller | --timeout=300s' 2026-03-20 16:26:25.699958 | controller | - ' ' 2026-03-20 16:26:25.699962 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.699966 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.699970 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.699975 | controller | - ' ' 2026-03-20 16:26:25.699979 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.699983 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.699987 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.699991 | controller | - ' ' 2026-03-20 16:26:25.699995 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.700000 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.700006 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.700018 | controller | available"' 2026-03-20 16:26:25.700022 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.700026 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.700031 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.700035 | controller | - ' exit 1' 2026-03-20 16:26:25.700039 | controller | - ' fi' 2026-03-20 16:26:25.700043 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.700048 | controller | - ' ' 2026-03-20 16:26:25.700052 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.700056 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.700060 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.700064 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.700068 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.700073 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.700077 | controller | -- \' 2026-03-20 16:26:25.700088 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.700093 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.700097 | controller | - ' logger.go:42: 16:09:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.700101 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.700105 | controller | - ' logger.go:42: 16:09:04 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.700111 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.700115 | controller | found' 2026-03-20 16:26:25.700120 | controller | - ' logger.go:42: 16:09:05 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.700124 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.700128 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.700132 | controller | - ' ' 2026-03-20 16:26:25.700136 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.700141 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.700147 | controller | - ' ' 2026-03-20 16:26:25.700151 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.700155 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.700159 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.700163 | controller | - ' exit 1' 2026-03-20 16:26:25.700168 | controller | - ' fi' 2026-03-20 16:26:25.700172 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.700176 | controller | - ' ' 2026-03-20 16:26:25.700180 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.700185 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.700189 | controller | | base64 -d)' 2026-03-20 16:26:25.700193 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.700197 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.700201 | controller | - ' exit 1' 2026-03-20 16:26:25.700205 | controller | - ' fi' 2026-03-20 16:26:25.700210 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.700214 | controller | - ' ' 2026-03-20 16:26:25.700218 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.700222 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.700226 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.700231 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.700235 | controller | --timeout=300s' 2026-03-20 16:26:25.700239 | controller | - ' ' 2026-03-20 16:26:25.700243 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.700247 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.700252 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.700256 | controller | - ' ' 2026-03-20 16:26:25.700260 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.700264 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.700276 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.700280 | controller | - ' ' 2026-03-20 16:26:25.700284 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.700288 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.700293 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.700297 | controller | available"' 2026-03-20 16:26:25.700302 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.700308 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.700312 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.700316 | controller | - ' exit 1' 2026-03-20 16:26:25.700322 | controller | - ' fi' 2026-03-20 16:26:25.700327 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.700331 | controller | - ' ' 2026-03-20 16:26:25.700335 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.700339 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.700345 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.700349 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.700354 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.700358 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.700362 | controller | -- \' 2026-03-20 16:26:25.700373 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.700377 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.700381 | controller | - ' logger.go:42: 16:09:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.700386 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.700397 | controller | - ' logger.go:42: 16:09:06 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.700401 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.700405 | controller | found' 2026-03-20 16:26:25.700409 | controller | - ' logger.go:42: 16:09:07 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.700414 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.700418 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.700422 | controller | - ' ' 2026-03-20 16:26:25.700426 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.700430 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.700434 | controller | - ' ' 2026-03-20 16:26:25.700439 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.700443 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.700447 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.700451 | controller | - ' exit 1' 2026-03-20 16:26:25.700455 | controller | - ' fi' 2026-03-20 16:26:25.700460 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.700464 | controller | - ' ' 2026-03-20 16:26:25.700468 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.700472 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.700476 | controller | | base64 -d)' 2026-03-20 16:26:25.700480 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.700485 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.700489 | controller | - ' exit 1' 2026-03-20 16:26:25.700493 | controller | - ' fi' 2026-03-20 16:26:25.700497 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.700501 | controller | - ' ' 2026-03-20 16:26:25.700505 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.700510 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.700514 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.700518 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.700522 | controller | --timeout=300s' 2026-03-20 16:26:25.700526 | controller | - ' ' 2026-03-20 16:26:25.700530 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.700534 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.700538 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.700544 | controller | - ' ' 2026-03-20 16:26:25.700549 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.700553 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.700557 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.700561 | controller | - ' ' 2026-03-20 16:26:25.700565 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.700569 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.700573 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.700578 | controller | available"' 2026-03-20 16:26:25.700582 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.700586 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.700590 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.700594 | controller | - ' exit 1' 2026-03-20 16:26:25.700599 | controller | - ' fi' 2026-03-20 16:26:25.700611 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.700615 | controller | - ' ' 2026-03-20 16:26:25.700619 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.700623 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.700627 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.700631 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.700636 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.700640 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.700646 | controller | -- \' 2026-03-20 16:26:25.700651 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.700661 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.700667 | controller | - ' logger.go:42: 16:09:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.700672 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.700676 | controller | - ' logger.go:42: 16:09:07 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.700680 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.700685 | controller | found' 2026-03-20 16:26:25.700689 | controller | - ' logger.go:42: 16:09:08 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.700709 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.700714 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.700718 | controller | - ' ' 2026-03-20 16:26:25.700722 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.700727 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.700731 | controller | - ' ' 2026-03-20 16:26:25.700735 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.700739 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.700743 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.700748 | controller | - ' exit 1' 2026-03-20 16:26:25.700752 | controller | - ' fi' 2026-03-20 16:26:25.700756 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.700762 | controller | - ' ' 2026-03-20 16:26:25.700767 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.700771 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.700775 | controller | | base64 -d)' 2026-03-20 16:26:25.700779 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.700784 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.700788 | controller | - ' exit 1' 2026-03-20 16:26:25.700792 | controller | - ' fi' 2026-03-20 16:26:25.700796 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.700801 | controller | - ' ' 2026-03-20 16:26:25.700805 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.700809 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.700815 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.700819 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.700824 | controller | --timeout=300s' 2026-03-20 16:26:25.700828 | controller | - ' ' 2026-03-20 16:26:25.700832 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.700836 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.700841 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.700845 | controller | - ' ' 2026-03-20 16:26:25.700849 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.700853 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.700858 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.700862 | controller | - ' ' 2026-03-20 16:26:25.700866 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.700870 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.700874 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.700878 | controller | available"' 2026-03-20 16:26:25.700883 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.700887 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.700891 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.700895 | controller | - ' exit 1' 2026-03-20 16:26:25.700899 | controller | - ' fi' 2026-03-20 16:26:25.700904 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.700908 | controller | - ' ' 2026-03-20 16:26:25.700912 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.700916 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.700920 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.700924 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.700929 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.700933 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.700937 | controller | -- \' 2026-03-20 16:26:25.700941 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.700955 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.700960 | controller | - ' logger.go:42: 16:09:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.700964 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.700968 | controller | - ' logger.go:42: 16:09:08 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.700973 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.700977 | controller | found' 2026-03-20 16:26:25.700981 | controller | - ' logger.go:42: 16:09:09 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.700985 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.700989 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.700996 | controller | - ' ' 2026-03-20 16:26:25.701000 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.701004 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.701008 | controller | - ' ' 2026-03-20 16:26:25.701013 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.701017 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.701021 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.701026 | controller | - ' exit 1' 2026-03-20 16:26:25.701030 | controller | - ' fi' 2026-03-20 16:26:25.701034 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.701038 | controller | - ' ' 2026-03-20 16:26:25.701044 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.701048 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.701053 | controller | | base64 -d)' 2026-03-20 16:26:25.701057 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.701061 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.701065 | controller | - ' exit 1' 2026-03-20 16:26:25.701070 | controller | - ' fi' 2026-03-20 16:26:25.701074 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.701078 | controller | - ' ' 2026-03-20 16:26:25.701082 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.701087 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.701091 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.701095 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.701099 | controller | --timeout=300s' 2026-03-20 16:26:25.701103 | controller | - ' ' 2026-03-20 16:26:25.701108 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.701112 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.701116 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.701120 | controller | - ' ' 2026-03-20 16:26:25.701125 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.701129 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.701133 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.701137 | controller | - ' ' 2026-03-20 16:26:25.701141 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.701147 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.701152 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.701156 | controller | available"' 2026-03-20 16:26:25.701160 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.701164 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.701169 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.701173 | controller | - ' exit 1' 2026-03-20 16:26:25.701177 | controller | - ' fi' 2026-03-20 16:26:25.701181 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.701186 | controller | - ' ' 2026-03-20 16:26:25.701190 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.701194 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.701198 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.701202 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.701207 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.701211 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.701215 | controller | -- \' 2026-03-20 16:26:25.701219 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.701224 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.701236 | controller | - ' logger.go:42: 16:09:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.701240 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.701244 | controller | - ' logger.go:42: 16:09:09 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.701249 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.701253 | controller | found' 2026-03-20 16:26:25.701264 | controller | - ' logger.go:42: 16:09:10 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.701269 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.701273 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.701277 | controller | - ' ' 2026-03-20 16:26:25.701284 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.701288 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.701292 | controller | - ' ' 2026-03-20 16:26:25.701296 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.701300 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.701305 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.701309 | controller | - ' exit 1' 2026-03-20 16:26:25.701313 | controller | - ' fi' 2026-03-20 16:26:25.701317 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.701321 | controller | - ' ' 2026-03-20 16:26:25.701326 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.701330 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.701334 | controller | | base64 -d)' 2026-03-20 16:26:25.701338 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.701342 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.701347 | controller | - ' exit 1' 2026-03-20 16:26:25.701352 | controller | - ' fi' 2026-03-20 16:26:25.701357 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.701363 | controller | - ' ' 2026-03-20 16:26:25.701367 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.701371 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.701375 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.701379 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.701383 | controller | --timeout=300s' 2026-03-20 16:26:25.701388 | controller | - ' ' 2026-03-20 16:26:25.701392 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.701396 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.701400 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.701404 | controller | - ' ' 2026-03-20 16:26:25.701409 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.701413 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.701417 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.701421 | controller | - ' ' 2026-03-20 16:26:25.701425 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.701430 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.701434 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.701438 | controller | available"' 2026-03-20 16:26:25.701442 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.701446 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.701450 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.701455 | controller | - ' exit 1' 2026-03-20 16:26:25.701459 | controller | - ' fi' 2026-03-20 16:26:25.701463 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.701467 | controller | - ' ' 2026-03-20 16:26:25.701472 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.701476 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.701480 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.701484 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.701488 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.701492 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.701497 | controller | -- \' 2026-03-20 16:26:25.701501 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.701505 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.701518 | controller | - ' logger.go:42: 16:09:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.701522 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.701527 | controller | - ' logger.go:42: 16:09:10 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.701531 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.701537 | controller | found' 2026-03-20 16:26:25.701541 | controller | - ' logger.go:42: 16:09:11 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.701546 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.701550 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.701554 | controller | - ' ' 2026-03-20 16:26:25.701558 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.701562 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.701567 | controller | - ' ' 2026-03-20 16:26:25.701571 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.701575 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.701579 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.701583 | controller | - ' exit 1' 2026-03-20 16:26:25.701588 | controller | - ' fi' 2026-03-20 16:26:25.701592 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.701596 | controller | - ' ' 2026-03-20 16:26:25.701600 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.701604 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.701609 | controller | | base64 -d)' 2026-03-20 16:26:25.701613 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.701617 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.701621 | controller | - ' exit 1' 2026-03-20 16:26:25.701626 | controller | - ' fi' 2026-03-20 16:26:25.701630 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.701634 | controller | - ' ' 2026-03-20 16:26:25.701638 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.701642 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.701646 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.701650 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.701655 | controller | --timeout=300s' 2026-03-20 16:26:25.701661 | controller | - ' ' 2026-03-20 16:26:25.701665 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.701669 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.701674 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.701678 | controller | - ' ' 2026-03-20 16:26:25.701682 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.701686 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.701690 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.701707 | controller | - ' ' 2026-03-20 16:26:25.701712 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.701716 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.701720 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.701725 | controller | available"' 2026-03-20 16:26:25.701729 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.701733 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.701737 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.701743 | controller | - ' exit 1' 2026-03-20 16:26:25.701748 | controller | - ' fi' 2026-03-20 16:26:25.701752 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.701756 | controller | - ' ' 2026-03-20 16:26:25.701760 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.701764 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.701769 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.701773 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.701777 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.701781 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.701785 | controller | -- \' 2026-03-20 16:26:25.701790 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.701794 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.701798 | controller | - ' logger.go:42: 16:09:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.701802 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.701815 | controller | - ' logger.go:42: 16:09:12 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.701820 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.701824 | controller | found' 2026-03-20 16:26:25.701829 | controller | - ' logger.go:42: 16:09:13 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.701833 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.701837 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.701841 | controller | - ' ' 2026-03-20 16:26:25.701845 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.701850 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.701854 | controller | - ' ' 2026-03-20 16:26:25.701858 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.701862 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.701866 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.701871 | controller | - ' exit 1' 2026-03-20 16:26:25.701877 | controller | - ' fi' 2026-03-20 16:26:25.701884 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.701888 | controller | - ' ' 2026-03-20 16:26:25.701892 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.701896 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.701901 | controller | | base64 -d)' 2026-03-20 16:26:25.701905 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.701909 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.701913 | controller | - ' exit 1' 2026-03-20 16:26:25.701917 | controller | - ' fi' 2026-03-20 16:26:25.701922 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.701926 | controller | - ' ' 2026-03-20 16:26:25.701930 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.701934 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.701938 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.701942 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.701948 | controller | --timeout=300s' 2026-03-20 16:26:25.701967 | controller | - ' ' 2026-03-20 16:26:25.701971 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.701975 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.701979 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.701984 | controller | - ' ' 2026-03-20 16:26:25.701988 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.701992 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.701996 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.702000 | controller | - ' ' 2026-03-20 16:26:25.702004 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.702009 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.702013 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.702017 | controller | available"' 2026-03-20 16:26:25.702021 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.702025 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.702029 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.702034 | controller | - ' exit 1' 2026-03-20 16:26:25.702038 | controller | - ' fi' 2026-03-20 16:26:25.702042 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.702046 | controller | - ' ' 2026-03-20 16:26:25.702050 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.702054 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.702059 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.702063 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.702067 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.702071 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.702075 | controller | -- \' 2026-03-20 16:26:25.702080 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.702084 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.702088 | controller | - ' logger.go:42: 16:09:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.702092 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.702104 | controller | - ' logger.go:42: 16:09:13 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.702108 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.702113 | controller | found' 2026-03-20 16:26:25.702117 | controller | - ' logger.go:42: 16:09:14 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.702121 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.702125 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.702130 | controller | - ' ' 2026-03-20 16:26:25.702134 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.702138 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.702144 | controller | - ' ' 2026-03-20 16:26:25.702148 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.702153 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.702157 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.702161 | controller | - ' exit 1' 2026-03-20 16:26:25.702165 | controller | - ' fi' 2026-03-20 16:26:25.702169 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.702173 | controller | - ' ' 2026-03-20 16:26:25.702178 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.702182 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.702186 | controller | | base64 -d)' 2026-03-20 16:26:25.702190 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.702194 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.702198 | controller | - ' exit 1' 2026-03-20 16:26:25.702203 | controller | - ' fi' 2026-03-20 16:26:25.702207 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.702211 | controller | - ' ' 2026-03-20 16:26:25.702215 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.702219 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.702223 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.702228 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.702232 | controller | --timeout=300s' 2026-03-20 16:26:25.702236 | controller | - ' ' 2026-03-20 16:26:25.702240 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.702244 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.702248 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.702253 | controller | - ' ' 2026-03-20 16:26:25.702257 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.702261 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.702265 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.702269 | controller | - ' ' 2026-03-20 16:26:25.702273 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.702278 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.702282 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.702286 | controller | available"' 2026-03-20 16:26:25.702290 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.702296 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.702300 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.702305 | controller | - ' exit 1' 2026-03-20 16:26:25.702309 | controller | - ' fi' 2026-03-20 16:26:25.702313 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.702317 | controller | - ' ' 2026-03-20 16:26:25.702321 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.702326 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.702330 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.702335 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.702340 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.702344 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.702348 | controller | -- \' 2026-03-20 16:26:25.702352 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.702357 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.702361 | controller | - ' logger.go:42: 16:09:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.702365 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.702369 | controller | - ' logger.go:42: 16:09:14 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.702380 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.702384 | controller | found' 2026-03-20 16:26:25.702389 | controller | - ' logger.go:42: 16:09:15 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.702400 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.702405 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.702409 | controller | - ' ' 2026-03-20 16:26:25.702413 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.702417 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.702422 | controller | - ' ' 2026-03-20 16:26:25.702435 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.702439 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.702444 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.702448 | controller | - ' exit 1' 2026-03-20 16:26:25.702452 | controller | - ' fi' 2026-03-20 16:26:25.702456 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.702460 | controller | - ' ' 2026-03-20 16:26:25.702465 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.702469 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.702473 | controller | | base64 -d)' 2026-03-20 16:26:25.702477 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.702481 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.702486 | controller | - ' exit 1' 2026-03-20 16:26:25.702490 | controller | - ' fi' 2026-03-20 16:26:25.702494 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.702498 | controller | - ' ' 2026-03-20 16:26:25.702502 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.702507 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.702511 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.702515 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.702519 | controller | --timeout=300s' 2026-03-20 16:26:25.702523 | controller | - ' ' 2026-03-20 16:26:25.702527 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.702531 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.702536 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.702540 | controller | - ' ' 2026-03-20 16:26:25.702546 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.702550 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.702554 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.702558 | controller | - ' ' 2026-03-20 16:26:25.702562 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.702566 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.702571 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.702575 | controller | available"' 2026-03-20 16:26:25.702579 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.702583 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.702587 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.702591 | controller | - ' exit 1' 2026-03-20 16:26:25.702596 | controller | - ' fi' 2026-03-20 16:26:25.702600 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.702604 | controller | - ' ' 2026-03-20 16:26:25.702608 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.702612 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.702616 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.702621 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.702625 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.702629 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.702633 | controller | -- \' 2026-03-20 16:26:25.702638 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.702642 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.702646 | controller | - ' logger.go:42: 16:09:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.702650 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.702654 | controller | - ' logger.go:42: 16:09:15 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.702665 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.702670 | controller | found' 2026-03-20 16:26:25.702674 | controller | - ' logger.go:42: 16:09:16 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.702678 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.702683 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.702687 | controller | - ' ' 2026-03-20 16:26:25.702702 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.702708 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.702713 | controller | - ' ' 2026-03-20 16:26:25.702717 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.702721 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.702725 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.702732 | controller | - ' exit 1' 2026-03-20 16:26:25.702736 | controller | - ' fi' 2026-03-20 16:26:25.702740 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.702745 | controller | - ' ' 2026-03-20 16:26:25.702750 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.702755 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.702759 | controller | | base64 -d)' 2026-03-20 16:26:25.702763 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.702767 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.702772 | controller | - ' exit 1' 2026-03-20 16:26:25.702776 | controller | - ' fi' 2026-03-20 16:26:25.702780 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.702784 | controller | - ' ' 2026-03-20 16:26:25.702788 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.702793 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.702797 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.702801 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.702805 | controller | --timeout=300s' 2026-03-20 16:26:25.702809 | controller | - ' ' 2026-03-20 16:26:25.702814 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.702818 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.702824 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.702828 | controller | - ' ' 2026-03-20 16:26:25.702832 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.702836 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.702840 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.702845 | controller | - ' ' 2026-03-20 16:26:25.702849 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.702853 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.702857 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.702861 | controller | available"' 2026-03-20 16:26:25.702865 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.702870 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.702874 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.702878 | controller | - ' exit 1' 2026-03-20 16:26:25.702882 | controller | - ' fi' 2026-03-20 16:26:25.702888 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.702892 | controller | - ' ' 2026-03-20 16:26:25.702897 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.702901 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.702905 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.702909 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.702913 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.702917 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.702922 | controller | -- \' 2026-03-20 16:26:25.702926 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.702932 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.702936 | controller | - ' logger.go:42: 16:09:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.702940 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.702944 | controller | - ' logger.go:42: 16:09:16 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.702948 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.702953 | controller | found' 2026-03-20 16:26:25.702965 | controller | - ' logger.go:42: 16:09:17 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.702969 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.702974 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.702978 | controller | - ' ' 2026-03-20 16:26:25.702982 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.702986 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.702990 | controller | - ' ' 2026-03-20 16:26:25.702995 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.703000 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.703005 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.703009 | controller | - ' exit 1' 2026-03-20 16:26:25.703013 | controller | - ' fi' 2026-03-20 16:26:25.703017 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.703023 | controller | - ' ' 2026-03-20 16:26:25.703027 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.703031 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.703036 | controller | | base64 -d)' 2026-03-20 16:26:25.703040 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.703044 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.703048 | controller | - ' exit 1' 2026-03-20 16:26:25.703053 | controller | - ' fi' 2026-03-20 16:26:25.703057 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.703061 | controller | - ' ' 2026-03-20 16:26:25.703065 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.703069 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.703074 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.703078 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.703082 | controller | --timeout=300s' 2026-03-20 16:26:25.703086 | controller | - ' ' 2026-03-20 16:26:25.703090 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.703094 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.703099 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.703103 | controller | - ' ' 2026-03-20 16:26:25.703107 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.703111 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.703115 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.703120 | controller | - ' ' 2026-03-20 16:26:25.703124 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.703130 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.703134 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.703138 | controller | available"' 2026-03-20 16:26:25.703143 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.703147 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.703151 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.703155 | controller | - ' exit 1' 2026-03-20 16:26:25.703159 | controller | - ' fi' 2026-03-20 16:26:25.703164 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.703168 | controller | - ' ' 2026-03-20 16:26:25.703172 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.703176 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.703180 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.703184 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.703189 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.703193 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.703197 | controller | -- \' 2026-03-20 16:26:25.703201 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.703205 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.703209 | controller | - ' logger.go:42: 16:09:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.703214 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.703218 | controller | - ' logger.go:42: 16:09:17 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.703222 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.703226 | controller | found' 2026-03-20 16:26:25.703230 | controller | - ' logger.go:42: 16:09:18 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.703242 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.703246 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.703251 | controller | - ' ' 2026-03-20 16:26:25.703255 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.703259 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.703263 | controller | - ' ' 2026-03-20 16:26:25.703267 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.703271 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.703276 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.703280 | controller | - ' exit 1' 2026-03-20 16:26:25.703284 | controller | - ' fi' 2026-03-20 16:26:25.703288 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.703293 | controller | - ' ' 2026-03-20 16:26:25.703298 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.703303 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.703307 | controller | | base64 -d)' 2026-03-20 16:26:25.703312 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.703317 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.703322 | controller | - ' exit 1' 2026-03-20 16:26:25.703327 | controller | - ' fi' 2026-03-20 16:26:25.703338 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.703342 | controller | - ' ' 2026-03-20 16:26:25.703347 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.703351 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.703355 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.703359 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.703363 | controller | --timeout=300s' 2026-03-20 16:26:25.703368 | controller | - ' ' 2026-03-20 16:26:25.703372 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.703376 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.703380 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.703384 | controller | - ' ' 2026-03-20 16:26:25.703388 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.703393 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.703397 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.703401 | controller | - ' ' 2026-03-20 16:26:25.703405 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.703409 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.703413 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.703418 | controller | available"' 2026-03-20 16:26:25.703422 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.703426 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.703430 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.703434 | controller | - ' exit 1' 2026-03-20 16:26:25.703439 | controller | - ' fi' 2026-03-20 16:26:25.703443 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.703447 | controller | - ' ' 2026-03-20 16:26:25.703451 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.703455 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.703459 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.703464 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.703468 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.703472 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.703476 | controller | -- \' 2026-03-20 16:26:25.703480 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.703485 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.703489 | controller | - ' logger.go:42: 16:09:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.703493 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.703497 | controller | - ' logger.go:42: 16:09:19 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.703501 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.703507 | controller | found' 2026-03-20 16:26:25.703512 | controller | - ' logger.go:42: 16:09:20 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.703516 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.703526 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.703531 | controller | - ' ' 2026-03-20 16:26:25.703535 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.703539 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.703544 | controller | - ' ' 2026-03-20 16:26:25.703548 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.703552 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.703556 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.703560 | controller | - ' exit 1' 2026-03-20 16:26:25.703565 | controller | - ' fi' 2026-03-20 16:26:25.703569 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.703573 | controller | - ' ' 2026-03-20 16:26:25.703579 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.703584 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.703588 | controller | | base64 -d)' 2026-03-20 16:26:25.703592 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.703596 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.703618 | controller | - ' exit 1' 2026-03-20 16:26:25.703624 | controller | - ' fi' 2026-03-20 16:26:25.703629 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.703634 | controller | - ' ' 2026-03-20 16:26:25.703639 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.703644 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.703651 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.703656 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.703661 | controller | --timeout=300s' 2026-03-20 16:26:25.703666 | controller | - ' ' 2026-03-20 16:26:25.703671 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.703676 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.703680 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.703685 | controller | - ' ' 2026-03-20 16:26:25.703690 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.703709 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.703714 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.703719 | controller | - ' ' 2026-03-20 16:26:25.703724 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.703729 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.703734 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.703739 | controller | available"' 2026-03-20 16:26:25.703744 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.703749 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.703754 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.703759 | controller | - ' exit 1' 2026-03-20 16:26:25.703766 | controller | - ' fi' 2026-03-20 16:26:25.703771 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.703776 | controller | - ' ' 2026-03-20 16:26:25.703781 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.703786 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.703791 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.703796 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.703803 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.703808 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.703813 | controller | -- \' 2026-03-20 16:26:25.703818 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.703823 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.703828 | controller | - ' logger.go:42: 16:09:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.703833 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.703838 | controller | - ' logger.go:42: 16:09:20 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.703842 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.703847 | controller | found' 2026-03-20 16:26:25.703852 | controller | - ' logger.go:42: 16:09:21 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.703857 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.703863 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.703867 | controller | - ' ' 2026-03-20 16:26:25.703879 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.703884 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.703888 | controller | - ' ' 2026-03-20 16:26:25.703893 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.703897 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.703901 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.703905 | controller | - ' exit 1' 2026-03-20 16:26:25.703909 | controller | - ' fi' 2026-03-20 16:26:25.703914 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.703918 | controller | - ' ' 2026-03-20 16:26:25.703922 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.703926 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.703931 | controller | | base64 -d)' 2026-03-20 16:26:25.703935 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.703939 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.703943 | controller | - ' exit 1' 2026-03-20 16:26:25.703947 | controller | - ' fi' 2026-03-20 16:26:25.703952 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.703956 | controller | - ' ' 2026-03-20 16:26:25.703960 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.703964 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.703969 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.703973 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.703977 | controller | --timeout=300s' 2026-03-20 16:26:25.703983 | controller | - ' ' 2026-03-20 16:26:25.703989 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.703994 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.703998 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.704002 | controller | - ' ' 2026-03-20 16:26:25.704006 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.704010 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.704015 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.704019 | controller | - ' ' 2026-03-20 16:26:25.704023 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.704027 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.704033 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.704038 | controller | available"' 2026-03-20 16:26:25.704042 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.704046 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.704050 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.704055 | controller | - ' exit 1' 2026-03-20 16:26:25.704059 | controller | - ' fi' 2026-03-20 16:26:25.704063 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.704067 | controller | - ' ' 2026-03-20 16:26:25.704071 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.704075 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.704079 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.704084 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.704088 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.704092 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.704096 | controller | -- \' 2026-03-20 16:26:25.704100 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.704104 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.704109 | controller | - ' logger.go:42: 16:09:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.704113 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.704117 | controller | - ' logger.go:42: 16:09:21 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.704121 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.704125 | controller | found' 2026-03-20 16:26:25.704130 | controller | - ' logger.go:42: 16:09:22 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.704134 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.704138 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.704142 | controller | - ' ' 2026-03-20 16:26:25.704153 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.704158 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.704162 | controller | - ' ' 2026-03-20 16:26:25.704168 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.704172 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.704176 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.704181 | controller | - ' exit 1' 2026-03-20 16:26:25.704185 | controller | - ' fi' 2026-03-20 16:26:25.704189 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.704193 | controller | - ' ' 2026-03-20 16:26:25.704197 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.704202 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.704206 | controller | | base64 -d)' 2026-03-20 16:26:25.704210 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.704214 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.704218 | controller | - ' exit 1' 2026-03-20 16:26:25.704222 | controller | - ' fi' 2026-03-20 16:26:25.704227 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.704231 | controller | - ' ' 2026-03-20 16:26:25.704235 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.704239 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.704244 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.704249 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.704254 | controller | --timeout=300s' 2026-03-20 16:26:25.704258 | controller | - ' ' 2026-03-20 16:26:25.704262 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.704266 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.704271 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.704275 | controller | - ' ' 2026-03-20 16:26:25.704279 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.704283 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.704288 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.704292 | controller | - ' ' 2026-03-20 16:26:25.704296 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.704300 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.704304 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.704309 | controller | available"' 2026-03-20 16:26:25.704313 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.704317 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.704321 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.704325 | controller | - ' exit 1' 2026-03-20 16:26:25.704330 | controller | - ' fi' 2026-03-20 16:26:25.704334 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.704338 | controller | - ' ' 2026-03-20 16:26:25.704342 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.704346 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.704350 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.704356 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.704360 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.704365 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.704369 | controller | -- \' 2026-03-20 16:26:25.704373 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.704377 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.704381 | controller | - ' logger.go:42: 16:09:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.704385 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.704390 | controller | - ' logger.go:42: 16:09:22 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.704394 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.704398 | controller | found' 2026-03-20 16:26:25.704402 | controller | - ' logger.go:42: 16:09:23 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.704406 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.704410 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.704415 | controller | - ' ' 2026-03-20 16:26:25.704419 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.704430 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.704434 | controller | - ' ' 2026-03-20 16:26:25.704438 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.704442 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.704447 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.704451 | controller | - ' exit 1' 2026-03-20 16:26:25.704455 | controller | - ' fi' 2026-03-20 16:26:25.704459 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.704464 | controller | - ' ' 2026-03-20 16:26:25.704468 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.704472 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.704476 | controller | | base64 -d)' 2026-03-20 16:26:25.704481 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.704485 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.704489 | controller | - ' exit 1' 2026-03-20 16:26:25.704493 | controller | - ' fi' 2026-03-20 16:26:25.704498 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.704502 | controller | - ' ' 2026-03-20 16:26:25.704506 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.704510 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.704514 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.704519 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.704523 | controller | --timeout=300s' 2026-03-20 16:26:25.704527 | controller | - ' ' 2026-03-20 16:26:25.704531 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.704535 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.704540 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.704544 | controller | - ' ' 2026-03-20 16:26:25.704548 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.704554 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.704558 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.704562 | controller | - ' ' 2026-03-20 16:26:25.704566 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.704570 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.704575 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.704579 | controller | available"' 2026-03-20 16:26:25.704583 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.704587 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.704591 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.704595 | controller | - ' exit 1' 2026-03-20 16:26:25.704600 | controller | - ' fi' 2026-03-20 16:26:25.704604 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.704608 | controller | - ' ' 2026-03-20 16:26:25.704612 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.704617 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.704621 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.704625 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.704629 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.704633 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.704638 | controller | -- \' 2026-03-20 16:26:25.704645 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.704649 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.704653 | controller | - ' logger.go:42: 16:09:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.704657 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.704664 | controller | - ' logger.go:42: 16:09:23 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.704668 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.704672 | controller | found' 2026-03-20 16:26:25.704676 | controller | - ' logger.go:42: 16:09:24 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.704681 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.704685 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.704689 | controller | - ' ' 2026-03-20 16:26:25.704706 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.704711 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.704722 | controller | - ' ' 2026-03-20 16:26:25.704726 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.704730 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.704735 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.704739 | controller | - ' exit 1' 2026-03-20 16:26:25.704743 | controller | - ' fi' 2026-03-20 16:26:25.704747 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.704751 | controller | - ' ' 2026-03-20 16:26:25.704756 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.704762 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.704766 | controller | | base64 -d)' 2026-03-20 16:26:25.704770 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.704775 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.704779 | controller | - ' exit 1' 2026-03-20 16:26:25.704783 | controller | - ' fi' 2026-03-20 16:26:25.704787 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.704792 | controller | - ' ' 2026-03-20 16:26:25.704796 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.704800 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.704804 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.704808 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.704812 | controller | --timeout=300s' 2026-03-20 16:26:25.704817 | controller | - ' ' 2026-03-20 16:26:25.704821 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.704825 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.704829 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.704833 | controller | - ' ' 2026-03-20 16:26:25.704837 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.704845 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.704849 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.704854 | controller | - ' ' 2026-03-20 16:26:25.704863 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.704868 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.704872 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.704876 | controller | available"' 2026-03-20 16:26:25.704880 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.704887 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.704891 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.704895 | controller | - ' exit 1' 2026-03-20 16:26:25.704899 | controller | - ' fi' 2026-03-20 16:26:25.704903 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.704908 | controller | - ' ' 2026-03-20 16:26:25.704912 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.704916 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.704920 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.704924 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.704929 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.704933 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.704937 | controller | -- \' 2026-03-20 16:26:25.704942 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.704946 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.704952 | controller | - ' logger.go:42: 16:09:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.704956 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.704960 | controller | - ' logger.go:42: 16:09:24 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.704964 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.704969 | controller | found' 2026-03-20 16:26:25.704974 | controller | - ' logger.go:42: 16:09:25 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.704978 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.704983 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.704987 | controller | - ' ' 2026-03-20 16:26:25.704991 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.704995 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.704999 | controller | - ' ' 2026-03-20 16:26:25.705011 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.705015 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.705020 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.705024 | controller | - ' exit 1' 2026-03-20 16:26:25.705028 | controller | - ' fi' 2026-03-20 16:26:25.705033 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.705037 | controller | - ' ' 2026-03-20 16:26:25.705041 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.705045 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.705050 | controller | | base64 -d)' 2026-03-20 16:26:25.705056 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.705060 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.705065 | controller | - ' exit 1' 2026-03-20 16:26:25.705069 | controller | - ' fi' 2026-03-20 16:26:25.705073 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.705077 | controller | - ' ' 2026-03-20 16:26:25.705082 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.705086 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.705090 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.705094 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.705098 | controller | --timeout=300s' 2026-03-20 16:26:25.705103 | controller | - ' ' 2026-03-20 16:26:25.705107 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.705111 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.705115 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.705119 | controller | - ' ' 2026-03-20 16:26:25.705124 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.705128 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.705132 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.705136 | controller | - ' ' 2026-03-20 16:26:25.705140 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.705145 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.705151 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.705155 | controller | available"' 2026-03-20 16:26:25.705159 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.705163 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.705167 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.705172 | controller | - ' exit 1' 2026-03-20 16:26:25.705176 | controller | - ' fi' 2026-03-20 16:26:25.705180 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.705184 | controller | - ' ' 2026-03-20 16:26:25.705189 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.705193 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.705197 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.705201 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.705205 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.705209 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.705214 | controller | -- \' 2026-03-20 16:26:25.705218 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.705222 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.705226 | controller | - ' logger.go:42: 16:09:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.705231 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.705235 | controller | - ' logger.go:42: 16:09:26 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.705239 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.705243 | controller | found' 2026-03-20 16:26:25.705249 | controller | - ' logger.go:42: 16:09:27 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.705253 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.705257 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.705261 | controller | - ' ' 2026-03-20 16:26:25.705266 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.705270 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.705274 | controller | - ' ' 2026-03-20 16:26:25.705285 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.705289 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.705293 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.705297 | controller | - ' exit 1' 2026-03-20 16:26:25.705302 | controller | - ' fi' 2026-03-20 16:26:25.705306 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.705310 | controller | - ' ' 2026-03-20 16:26:25.705315 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.705319 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.705323 | controller | | base64 -d)' 2026-03-20 16:26:25.705327 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.705331 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.705335 | controller | - ' exit 1' 2026-03-20 16:26:25.705339 | controller | - ' fi' 2026-03-20 16:26:25.705344 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.705351 | controller | - ' ' 2026-03-20 16:26:25.705356 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.705360 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.705364 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.705368 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.705372 | controller | --timeout=300s' 2026-03-20 16:26:25.705376 | controller | - ' ' 2026-03-20 16:26:25.705381 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.705385 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.705389 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.705393 | controller | - ' ' 2026-03-20 16:26:25.705397 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.705402 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.705406 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.705410 | controller | - ' ' 2026-03-20 16:26:25.705414 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.705418 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.705424 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.705428 | controller | available"' 2026-03-20 16:26:25.705433 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.705437 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.705441 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.705445 | controller | - ' exit 1' 2026-03-20 16:26:25.705450 | controller | - ' fi' 2026-03-20 16:26:25.705454 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.705458 | controller | - ' ' 2026-03-20 16:26:25.705462 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.705466 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.705470 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.705475 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.705479 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.705483 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.705487 | controller | -- \' 2026-03-20 16:26:25.705492 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.705496 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.705500 | controller | - ' logger.go:42: 16:09:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.705504 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.705508 | controller | - ' logger.go:42: 16:09:27 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.705513 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.705517 | controller | found' 2026-03-20 16:26:25.705521 | controller | - ' logger.go:42: 16:09:28 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.705527 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.705531 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.705535 | controller | - ' ' 2026-03-20 16:26:25.705540 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.705544 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.705548 | controller | - ' ' 2026-03-20 16:26:25.705552 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.705556 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.705567 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.705572 | controller | - ' exit 1' 2026-03-20 16:26:25.705576 | controller | - ' fi' 2026-03-20 16:26:25.705580 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.705585 | controller | - ' ' 2026-03-20 16:26:25.705589 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.705593 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.705597 | controller | | base64 -d)' 2026-03-20 16:26:25.705602 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.705606 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.705610 | controller | - ' exit 1' 2026-03-20 16:26:25.705614 | controller | - ' fi' 2026-03-20 16:26:25.705619 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.705623 | controller | - ' ' 2026-03-20 16:26:25.705627 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.705631 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.705635 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.705639 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.705644 | controller | --timeout=300s' 2026-03-20 16:26:25.705650 | controller | - ' ' 2026-03-20 16:26:25.705654 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.705658 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.705662 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.705667 | controller | - ' ' 2026-03-20 16:26:25.705671 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.705675 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.705679 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.705683 | controller | - ' ' 2026-03-20 16:26:25.705687 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.705702 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.705709 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.705713 | controller | available"' 2026-03-20 16:26:25.705717 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.705721 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.705726 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.705730 | controller | - ' exit 1' 2026-03-20 16:26:25.705734 | controller | - ' fi' 2026-03-20 16:26:25.705740 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.705744 | controller | - ' ' 2026-03-20 16:26:25.705749 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.705753 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.705757 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.705761 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.705765 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.705770 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.705774 | controller | -- \' 2026-03-20 16:26:25.705778 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.705782 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.705786 | controller | - ' logger.go:42: 16:09:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.705791 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.705797 | controller | - ' logger.go:42: 16:09:28 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.705801 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.705805 | controller | found' 2026-03-20 16:26:25.705810 | controller | - ' logger.go:42: 16:09:29 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.705814 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.705818 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.705822 | controller | - ' ' 2026-03-20 16:26:25.705826 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.705831 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.705835 | controller | - ' ' 2026-03-20 16:26:25.705839 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.705843 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.705855 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.705860 | controller | - ' exit 1' 2026-03-20 16:26:25.705864 | controller | - ' fi' 2026-03-20 16:26:25.705868 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.705873 | controller | - ' ' 2026-03-20 16:26:25.705877 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.705881 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.705885 | controller | | base64 -d)' 2026-03-20 16:26:25.705889 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.705893 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.705898 | controller | - ' exit 1' 2026-03-20 16:26:25.705902 | controller | - ' fi' 2026-03-20 16:26:25.705906 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.705910 | controller | - ' ' 2026-03-20 16:26:25.705914 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.705919 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.705923 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.705927 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.705931 | controller | --timeout=300s' 2026-03-20 16:26:25.705935 | controller | - ' ' 2026-03-20 16:26:25.705941 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.705946 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.705950 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.705954 | controller | - ' ' 2026-03-20 16:26:25.705958 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.705962 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.705967 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.705971 | controller | - ' ' 2026-03-20 16:26:25.705975 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.705979 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.705983 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.705988 | controller | available"' 2026-03-20 16:26:25.705992 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.705998 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.706002 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.706006 | controller | - ' exit 1' 2026-03-20 16:26:25.706010 | controller | - ' fi' 2026-03-20 16:26:25.706014 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.706019 | controller | - ' ' 2026-03-20 16:26:25.706023 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.706027 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.706031 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.706035 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.706040 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.706044 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.706048 | controller | -- \' 2026-03-20 16:26:25.706052 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.706056 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.706060 | controller | - ' logger.go:42: 16:09:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.706065 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.706069 | controller | - ' logger.go:42: 16:09:29 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.706073 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.706077 | controller | found' 2026-03-20 16:26:25.706081 | controller | - ' logger.go:42: 16:09:30 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.706085 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.706090 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.706094 | controller | - ' ' 2026-03-20 16:26:25.706101 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.706105 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.706110 | controller | - ' ' 2026-03-20 16:26:25.706114 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.706120 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.706124 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.706128 | controller | - ' exit 1' 2026-03-20 16:26:25.706140 | controller | - ' fi' 2026-03-20 16:26:25.706145 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.706149 | controller | - ' ' 2026-03-20 16:26:25.706156 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.706160 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.706164 | controller | | base64 -d)' 2026-03-20 16:26:25.706168 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.706172 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.706177 | controller | - ' exit 1' 2026-03-20 16:26:25.706181 | controller | - ' fi' 2026-03-20 16:26:25.706185 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.706189 | controller | - ' ' 2026-03-20 16:26:25.706194 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.706198 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.706202 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.706206 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.706210 | controller | --timeout=300s' 2026-03-20 16:26:25.706215 | controller | - ' ' 2026-03-20 16:26:25.706219 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.706223 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.706227 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.706231 | controller | - ' ' 2026-03-20 16:26:25.706235 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.706240 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.706244 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.706248 | controller | - ' ' 2026-03-20 16:26:25.706252 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.706256 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.706260 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.706265 | controller | available"' 2026-03-20 16:26:25.706269 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.706273 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.706277 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.706281 | controller | - ' exit 1' 2026-03-20 16:26:25.706286 | controller | - ' fi' 2026-03-20 16:26:25.706290 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.706294 | controller | - ' ' 2026-03-20 16:26:25.706298 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.706302 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.706307 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.706311 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.706317 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.706321 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.706325 | controller | -- \' 2026-03-20 16:26:25.706329 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.706333 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.706338 | controller | - ' logger.go:42: 16:09:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.706342 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.706346 | controller | - ' logger.go:42: 16:09:30 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.706350 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.706354 | controller | found' 2026-03-20 16:26:25.706359 | controller | - ' logger.go:42: 16:09:31 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.706363 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.706367 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.706371 | controller | - ' ' 2026-03-20 16:26:25.706375 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.706379 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.706384 | controller | - ' ' 2026-03-20 16:26:25.706388 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.706392 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.706396 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.706400 | controller | - ' exit 1' 2026-03-20 16:26:25.706404 | controller | - ' fi' 2026-03-20 16:26:25.706415 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.706420 | controller | - ' ' 2026-03-20 16:26:25.706424 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.706428 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.706432 | controller | | base64 -d)' 2026-03-20 16:26:25.706437 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.706441 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.706445 | controller | - ' exit 1' 2026-03-20 16:26:25.706449 | controller | - ' fi' 2026-03-20 16:26:25.706453 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.706458 | controller | - ' ' 2026-03-20 16:26:25.706462 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.706466 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.706470 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.706474 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.706479 | controller | --timeout=300s' 2026-03-20 16:26:25.706483 | controller | - ' ' 2026-03-20 16:26:25.706487 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.706491 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.706495 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.706500 | controller | - ' ' 2026-03-20 16:26:25.706504 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.706510 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.706514 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.706518 | controller | - ' ' 2026-03-20 16:26:25.706522 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.706527 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.706531 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.706535 | controller | available"' 2026-03-20 16:26:25.706539 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.706543 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.706548 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.706552 | controller | - ' exit 1' 2026-03-20 16:26:25.706556 | controller | - ' fi' 2026-03-20 16:26:25.706560 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.706564 | controller | - ' ' 2026-03-20 16:26:25.706570 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.706575 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.706579 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.706583 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.706587 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.706592 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.706596 | controller | -- \' 2026-03-20 16:26:25.706600 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.706604 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.706609 | controller | - ' logger.go:42: 16:09:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.706614 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.706619 | controller | - ' logger.go:42: 16:09:31 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.706623 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.706627 | controller | found' 2026-03-20 16:26:25.706631 | controller | - ' logger.go:42: 16:09:32 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.706636 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.706640 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.706644 | controller | - ' ' 2026-03-20 16:26:25.706648 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.706652 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.706656 | controller | - ' ' 2026-03-20 16:26:25.706661 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.706665 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.706669 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.706675 | controller | - ' exit 1' 2026-03-20 16:26:25.706679 | controller | - ' fi' 2026-03-20 16:26:25.706683 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.706688 | controller | - ' ' 2026-03-20 16:26:25.706709 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.706716 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.706722 | controller | | base64 -d)' 2026-03-20 16:26:25.706726 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.706731 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.706735 | controller | - ' exit 1' 2026-03-20 16:26:25.706739 | controller | - ' fi' 2026-03-20 16:26:25.706743 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.706747 | controller | - ' ' 2026-03-20 16:26:25.706752 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.706756 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.706760 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.706764 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.706769 | controller | --timeout=300s' 2026-03-20 16:26:25.706773 | controller | - ' ' 2026-03-20 16:26:25.706777 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.706781 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.706785 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.706790 | controller | - ' ' 2026-03-20 16:26:25.706794 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.706798 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.706804 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.706808 | controller | - ' ' 2026-03-20 16:26:25.706813 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.706817 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.706821 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.706825 | controller | available"' 2026-03-20 16:26:25.706829 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.706833 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.706838 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.706842 | controller | - ' exit 1' 2026-03-20 16:26:25.706846 | controller | - ' fi' 2026-03-20 16:26:25.706850 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.706854 | controller | - ' ' 2026-03-20 16:26:25.706859 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.706863 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.706867 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.706871 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.706875 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.706880 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.706884 | controller | -- \' 2026-03-20 16:26:25.706888 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.706892 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.706898 | controller | - ' logger.go:42: 16:09:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.706903 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.706907 | controller | - ' logger.go:42: 16:09:33 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.706911 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.706915 | controller | found' 2026-03-20 16:26:25.706920 | controller | - ' logger.go:42: 16:09:34 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.706924 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.706928 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.706932 | controller | - ' ' 2026-03-20 16:26:25.706936 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.706941 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.706945 | controller | - ' ' 2026-03-20 16:26:25.706949 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.706953 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.706957 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.706962 | controller | - ' exit 1' 2026-03-20 16:26:25.706966 | controller | - ' fi' 2026-03-20 16:26:25.706970 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.706974 | controller | - ' ' 2026-03-20 16:26:25.706978 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.706991 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.706996 | controller | | base64 -d)' 2026-03-20 16:26:25.707000 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.707004 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.707008 | controller | - ' exit 1' 2026-03-20 16:26:25.707012 | controller | - ' fi' 2026-03-20 16:26:25.707017 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.707021 | controller | - ' ' 2026-03-20 16:26:25.707025 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.707029 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.707033 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.707037 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.707042 | controller | --timeout=300s' 2026-03-20 16:26:25.707046 | controller | - ' ' 2026-03-20 16:26:25.707050 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.707054 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.707058 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.707062 | controller | - ' ' 2026-03-20 16:26:25.707067 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.707071 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.707075 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.707079 | controller | - ' ' 2026-03-20 16:26:25.707086 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.707090 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.707094 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.707100 | controller | available"' 2026-03-20 16:26:25.707104 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.707108 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.707113 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.707117 | controller | - ' exit 1' 2026-03-20 16:26:25.707121 | controller | - ' fi' 2026-03-20 16:26:25.707125 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.707129 | controller | - ' ' 2026-03-20 16:26:25.707133 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.707137 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.707142 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.707147 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.707152 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.707156 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.707160 | controller | -- \' 2026-03-20 16:26:25.707164 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.707168 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.707187 | controller | - ' logger.go:42: 16:09:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.707191 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.707196 | controller | - ' logger.go:42: 16:09:34 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.707200 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.707204 | controller | found' 2026-03-20 16:26:25.707208 | controller | - ' logger.go:42: 16:09:35 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.707212 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.707216 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.707221 | controller | - ' ' 2026-03-20 16:26:25.707225 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.707229 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.707233 | controller | - ' ' 2026-03-20 16:26:25.707237 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.707241 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.707246 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.707250 | controller | - ' exit 1' 2026-03-20 16:26:25.707254 | controller | - ' fi' 2026-03-20 16:26:25.707266 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.707270 | controller | - ' ' 2026-03-20 16:26:25.707274 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.707285 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.707290 | controller | | base64 -d)' 2026-03-20 16:26:25.707294 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.707298 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.707302 | controller | - ' exit 1' 2026-03-20 16:26:25.707307 | controller | - ' fi' 2026-03-20 16:26:25.707311 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.707315 | controller | - ' ' 2026-03-20 16:26:25.707319 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.707333 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.707337 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.707342 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.707346 | controller | --timeout=300s' 2026-03-20 16:26:25.707350 | controller | - ' ' 2026-03-20 16:26:25.707354 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.707358 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.707363 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.707367 | controller | - ' ' 2026-03-20 16:26:25.707371 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.707375 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.707380 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.707384 | controller | - ' ' 2026-03-20 16:26:25.707395 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.707399 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.707403 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.707408 | controller | available"' 2026-03-20 16:26:25.707412 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.707419 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.707424 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.707428 | controller | - ' exit 1' 2026-03-20 16:26:25.707432 | controller | - ' fi' 2026-03-20 16:26:25.707436 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.707441 | controller | - ' ' 2026-03-20 16:26:25.707452 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.707456 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.707461 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.707465 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.707469 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.707473 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.707477 | controller | -- \' 2026-03-20 16:26:25.707482 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.707486 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.707490 | controller | - ' logger.go:42: 16:09:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.707494 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.707498 | controller | - ' logger.go:42: 16:09:35 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.707502 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.707507 | controller | found' 2026-03-20 16:26:25.707511 | controller | - ' logger.go:42: 16:09:36 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.707517 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.707521 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.707525 | controller | - ' ' 2026-03-20 16:26:25.707530 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.707534 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.707538 | controller | - ' ' 2026-03-20 16:26:25.707542 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.707546 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.707551 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.707555 | controller | - ' exit 1' 2026-03-20 16:26:25.707559 | controller | - ' fi' 2026-03-20 16:26:25.707563 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.707567 | controller | - ' ' 2026-03-20 16:26:25.707571 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.707576 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.707586 | controller | | base64 -d)' 2026-03-20 16:26:25.707591 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.707595 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.707599 | controller | - ' exit 1' 2026-03-20 16:26:25.707604 | controller | - ' fi' 2026-03-20 16:26:25.707626 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.707630 | controller | - ' ' 2026-03-20 16:26:25.707635 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.707639 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.707643 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.707647 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.707651 | controller | --timeout=300s' 2026-03-20 16:26:25.707655 | controller | - ' ' 2026-03-20 16:26:25.707660 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.707664 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.707668 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.707672 | controller | - ' ' 2026-03-20 16:26:25.707676 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.707681 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.707685 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.707689 | controller | - ' ' 2026-03-20 16:26:25.707708 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.707714 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.707718 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.707725 | controller | available"' 2026-03-20 16:26:25.707729 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.707733 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.707737 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.707742 | controller | - ' exit 1' 2026-03-20 16:26:25.707746 | controller | - ' fi' 2026-03-20 16:26:25.707750 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.707756 | controller | - ' ' 2026-03-20 16:26:25.707760 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.707764 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.707769 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.707773 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.707777 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.707781 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.707785 | controller | -- \' 2026-03-20 16:26:25.707790 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.707795 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.707801 | controller | - ' logger.go:42: 16:09:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.707806 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.707810 | controller | - ' logger.go:42: 16:09:36 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.707814 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.707818 | controller | found' 2026-03-20 16:26:25.707822 | controller | - ' logger.go:42: 16:09:37 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.707827 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.707831 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.707835 | controller | - ' ' 2026-03-20 16:26:25.707839 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.707844 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.707848 | controller | - ' ' 2026-03-20 16:26:25.707852 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.707856 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.707860 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.707865 | controller | - ' exit 1' 2026-03-20 16:26:25.707869 | controller | - ' fi' 2026-03-20 16:26:25.707873 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.707877 | controller | - ' ' 2026-03-20 16:26:25.707881 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.707886 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.707890 | controller | | base64 -d)' 2026-03-20 16:26:25.707904 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.707909 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.707913 | controller | - ' exit 1' 2026-03-20 16:26:25.707917 | controller | - ' fi' 2026-03-20 16:26:25.707922 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.707926 | controller | - ' ' 2026-03-20 16:26:25.707930 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.707934 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.707939 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.707943 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.707947 | controller | --timeout=300s' 2026-03-20 16:26:25.707951 | controller | - ' ' 2026-03-20 16:26:25.707956 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.707962 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.707966 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.707970 | controller | - ' ' 2026-03-20 16:26:25.707975 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.707979 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.707983 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.707987 | controller | - ' ' 2026-03-20 16:26:25.707991 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.707995 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.708001 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.708006 | controller | available"' 2026-03-20 16:26:25.708010 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.708014 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.708018 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.708022 | controller | - ' exit 1' 2026-03-20 16:26:25.708027 | controller | - ' fi' 2026-03-20 16:26:25.708031 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.708035 | controller | - ' ' 2026-03-20 16:26:25.708039 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.708043 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.708048 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.708052 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.708056 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.708060 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.708064 | controller | -- \' 2026-03-20 16:26:25.708068 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.708073 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.708077 | controller | - ' logger.go:42: 16:09:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.708081 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.708085 | controller | - ' logger.go:42: 16:09:37 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.708089 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.708094 | controller | found' 2026-03-20 16:26:25.708098 | controller | - ' logger.go:42: 16:09:38 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.708102 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.708106 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.708112 | controller | - ' ' 2026-03-20 16:26:25.708116 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.708121 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.708125 | controller | - ' ' 2026-03-20 16:26:25.708129 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.708133 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.708139 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.708144 | controller | - ' exit 1' 2026-03-20 16:26:25.708148 | controller | - ' fi' 2026-03-20 16:26:25.708152 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.708156 | controller | - ' ' 2026-03-20 16:26:25.708161 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.708165 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.708169 | controller | | base64 -d)' 2026-03-20 16:26:25.708174 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.708185 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.708189 | controller | - ' exit 1' 2026-03-20 16:26:25.708194 | controller | - ' fi' 2026-03-20 16:26:25.708200 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.708205 | controller | - ' ' 2026-03-20 16:26:25.708209 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.708213 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.708217 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.708221 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.708226 | controller | --timeout=300s' 2026-03-20 16:26:25.708230 | controller | - ' ' 2026-03-20 16:26:25.708234 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.708238 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.708243 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.708247 | controller | - ' ' 2026-03-20 16:26:25.708251 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.708255 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.708260 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.708264 | controller | - ' ' 2026-03-20 16:26:25.708268 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.708272 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.708276 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.708281 | controller | available"' 2026-03-20 16:26:25.708285 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.708289 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.708294 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.708298 | controller | - ' exit 1' 2026-03-20 16:26:25.708302 | controller | - ' fi' 2026-03-20 16:26:25.708306 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.708311 | controller | - ' ' 2026-03-20 16:26:25.708315 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.708319 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.708323 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.708327 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.708332 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.708338 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.708342 | controller | -- \' 2026-03-20 16:26:25.708346 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.708350 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.708355 | controller | - ' logger.go:42: 16:09:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.708359 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.708365 | controller | - ' logger.go:42: 16:09:38 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.708369 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.708373 | controller | found' 2026-03-20 16:26:25.708378 | controller | - ' logger.go:42: 16:09:39 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.708382 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.708386 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.708390 | controller | - ' ' 2026-03-20 16:26:25.708395 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.708399 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.708403 | controller | - ' ' 2026-03-20 16:26:25.708407 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.708411 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.708416 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.708420 | controller | - ' exit 1' 2026-03-20 16:26:25.708424 | controller | - ' fi' 2026-03-20 16:26:25.708428 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.708433 | controller | - ' ' 2026-03-20 16:26:25.708437 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.708441 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.708445 | controller | | base64 -d)' 2026-03-20 16:26:25.708450 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.708461 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.708465 | controller | - ' exit 1' 2026-03-20 16:26:25.708470 | controller | - ' fi' 2026-03-20 16:26:25.708474 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.708478 | controller | - ' ' 2026-03-20 16:26:25.708483 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.708487 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.708491 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.708495 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.708499 | controller | --timeout=300s' 2026-03-20 16:26:25.708504 | controller | - ' ' 2026-03-20 16:26:25.708508 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.708512 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.708516 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.708520 | controller | - ' ' 2026-03-20 16:26:25.708525 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.708529 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.708535 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.708539 | controller | - ' ' 2026-03-20 16:26:25.708543 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.708548 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.708552 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.708556 | controller | available"' 2026-03-20 16:26:25.708560 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.708565 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.708569 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.708573 | controller | - ' exit 1' 2026-03-20 16:26:25.708577 | controller | - ' fi' 2026-03-20 16:26:25.708581 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.708585 | controller | - ' ' 2026-03-20 16:26:25.708590 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.708594 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.708598 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.708602 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.708606 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.708611 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.708615 | controller | -- \' 2026-03-20 16:26:25.708619 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.708623 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.708628 | controller | - ' logger.go:42: 16:09:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.708632 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.708636 | controller | - ' logger.go:42: 16:09:40 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.708640 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.708644 | controller | found' 2026-03-20 16:26:25.708649 | controller | - ' logger.go:42: 16:09:41 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.708653 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.708657 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.708661 | controller | - ' ' 2026-03-20 16:26:25.708666 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.708670 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.708674 | controller | - ' ' 2026-03-20 16:26:25.708678 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.708683 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.708687 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.708705 | controller | - ' exit 1' 2026-03-20 16:26:25.708712 | controller | - ' fi' 2026-03-20 16:26:25.708716 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.708721 | controller | - ' ' 2026-03-20 16:26:25.708727 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.708731 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.708739 | controller | | base64 -d)' 2026-03-20 16:26:25.708743 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.708747 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.708751 | controller | - ' exit 1' 2026-03-20 16:26:25.708756 | controller | - ' fi' 2026-03-20 16:26:25.708767 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.708771 | controller | - ' ' 2026-03-20 16:26:25.708776 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.708780 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.708784 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.708788 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.708793 | controller | --timeout=300s' 2026-03-20 16:26:25.708797 | controller | - ' ' 2026-03-20 16:26:25.708801 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.708805 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.708809 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.708814 | controller | - ' ' 2026-03-20 16:26:25.708818 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.708822 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.708826 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.708830 | controller | - ' ' 2026-03-20 16:26:25.708834 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.708841 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.708845 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.708849 | controller | available"' 2026-03-20 16:26:25.708853 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.708857 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.708862 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.708866 | controller | - ' exit 1' 2026-03-20 16:26:25.708870 | controller | - ' fi' 2026-03-20 16:26:25.708874 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.708878 | controller | - ' ' 2026-03-20 16:26:25.708883 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.708887 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.708891 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.708895 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.708900 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.708904 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.708908 | controller | -- \' 2026-03-20 16:26:25.708912 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.708916 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.708920 | controller | - ' logger.go:42: 16:09:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.708926 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.708931 | controller | - ' logger.go:42: 16:09:41 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.708936 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.708942 | controller | found' 2026-03-20 16:26:25.708947 | controller | - ' logger.go:42: 16:09:42 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.708953 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.708959 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.708964 | controller | - ' ' 2026-03-20 16:26:25.708970 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.708976 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.708982 | controller | - ' ' 2026-03-20 16:26:25.708987 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.708993 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.708999 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.709004 | controller | - ' exit 1' 2026-03-20 16:26:25.709009 | controller | - ' fi' 2026-03-20 16:26:25.709013 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.709017 | controller | - ' ' 2026-03-20 16:26:25.709021 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.709025 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.709030 | controller | | base64 -d)' 2026-03-20 16:26:25.709034 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.709038 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.709042 | controller | - ' exit 1' 2026-03-20 16:26:25.709046 | controller | - ' fi' 2026-03-20 16:26:25.709059 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.709064 | controller | - ' ' 2026-03-20 16:26:25.709068 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.709072 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.709076 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.709081 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.709085 | controller | --timeout=300s' 2026-03-20 16:26:25.709089 | controller | - ' ' 2026-03-20 16:26:25.709093 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.709097 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.709102 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.709106 | controller | - ' ' 2026-03-20 16:26:25.709110 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.709114 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.709118 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.709123 | controller | - ' ' 2026-03-20 16:26:25.709127 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.709131 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.709135 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.709142 | controller | available"' 2026-03-20 16:26:25.709146 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.709150 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.709155 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.709159 | controller | - ' exit 1' 2026-03-20 16:26:25.709163 | controller | - ' fi' 2026-03-20 16:26:25.709167 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.709171 | controller | - ' ' 2026-03-20 16:26:25.709176 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.709180 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.709184 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.709188 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.709192 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.709197 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.709201 | controller | -- \' 2026-03-20 16:26:25.709205 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.709209 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.709214 | controller | - ' logger.go:42: 16:09:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.709218 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.709222 | controller | - ' logger.go:42: 16:09:42 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.709226 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.709231 | controller | found' 2026-03-20 16:26:25.709235 | controller | - ' logger.go:42: 16:09:43 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.709239 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.709243 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.709247 | controller | - ' ' 2026-03-20 16:26:25.709252 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.709256 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.709260 | controller | - ' ' 2026-03-20 16:26:25.709264 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.709269 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.709273 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.709277 | controller | - ' exit 1' 2026-03-20 16:26:25.709284 | controller | - ' fi' 2026-03-20 16:26:25.709288 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.709292 | controller | - ' ' 2026-03-20 16:26:25.709296 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.709301 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.709305 | controller | | base64 -d)' 2026-03-20 16:26:25.709309 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.709313 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.709317 | controller | - ' exit 1' 2026-03-20 16:26:25.709322 | controller | - ' fi' 2026-03-20 16:26:25.709326 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.709332 | controller | - ' ' 2026-03-20 16:26:25.709343 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.709349 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.709354 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.709358 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.709362 | controller | --timeout=300s' 2026-03-20 16:26:25.709366 | controller | - ' ' 2026-03-20 16:26:25.709371 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.709375 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.709379 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.709383 | controller | - ' ' 2026-03-20 16:26:25.709388 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.709392 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.709396 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.709400 | controller | - ' ' 2026-03-20 16:26:25.709404 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.709409 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.709413 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.709417 | controller | available"' 2026-03-20 16:26:25.709421 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.709425 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.709430 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.709434 | controller | - ' exit 1' 2026-03-20 16:26:25.709438 | controller | - ' fi' 2026-03-20 16:26:25.709442 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.709446 | controller | - ' ' 2026-03-20 16:26:25.709450 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.709455 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.709459 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.709463 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.709467 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.709472 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.709476 | controller | -- \' 2026-03-20 16:26:25.709480 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.709484 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.709488 | controller | - ' logger.go:42: 16:09:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.709493 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.709497 | controller | - ' logger.go:42: 16:09:43 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.709501 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.709505 | controller | found' 2026-03-20 16:26:25.709511 | controller | - ' logger.go:42: 16:09:44 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.709515 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.709521 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.709526 | controller | - ' ' 2026-03-20 16:26:25.709530 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.709534 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.709538 | controller | - ' ' 2026-03-20 16:26:25.709542 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.709547 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.709551 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.709555 | controller | - ' exit 1' 2026-03-20 16:26:25.709559 | controller | - ' fi' 2026-03-20 16:26:25.709563 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.709568 | controller | - ' ' 2026-03-20 16:26:25.709572 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.709576 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.709580 | controller | | base64 -d)' 2026-03-20 16:26:25.709584 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.709589 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.709593 | controller | - ' exit 1' 2026-03-20 16:26:25.709597 | controller | - ' fi' 2026-03-20 16:26:25.709601 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.709606 | controller | - ' ' 2026-03-20 16:26:25.709610 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.709621 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.709626 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.709630 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.709634 | controller | --timeout=300s' 2026-03-20 16:26:25.709640 | controller | - ' ' 2026-03-20 16:26:25.709645 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.709649 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.709653 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.709657 | controller | - ' ' 2026-03-20 16:26:25.709662 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.709666 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.709670 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.709674 | controller | - ' ' 2026-03-20 16:26:25.709678 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.709683 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.709687 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.709703 | controller | available"' 2026-03-20 16:26:25.709710 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.709714 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.709718 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.709723 | controller | - ' exit 1' 2026-03-20 16:26:25.709727 | controller | - ' fi' 2026-03-20 16:26:25.709731 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.709735 | controller | - ' ' 2026-03-20 16:26:25.709741 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.709746 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.709750 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.709754 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.709758 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.709763 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.709767 | controller | -- \' 2026-03-20 16:26:25.709771 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.709775 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.709779 | controller | - ' logger.go:42: 16:09:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.709784 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.709788 | controller | - ' logger.go:42: 16:09:44 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.709792 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.709796 | controller | found' 2026-03-20 16:26:25.709801 | controller | - ' logger.go:42: 16:09:45 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.709805 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.709809 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.709813 | controller | - ' ' 2026-03-20 16:26:25.709818 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.709822 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.709826 | controller | - ' ' 2026-03-20 16:26:25.709830 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.709834 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.709838 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.709843 | controller | - ' exit 1' 2026-03-20 16:26:25.709847 | controller | - ' fi' 2026-03-20 16:26:25.709851 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.709855 | controller | - ' ' 2026-03-20 16:26:25.709860 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.709864 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.709868 | controller | | base64 -d)' 2026-03-20 16:26:25.709872 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.709876 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.709881 | controller | - ' exit 1' 2026-03-20 16:26:25.709885 | controller | - ' fi' 2026-03-20 16:26:25.709889 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.709893 | controller | - ' ' 2026-03-20 16:26:25.709898 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.709909 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.709914 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.709918 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.709922 | controller | --timeout=300s' 2026-03-20 16:26:25.709926 | controller | - ' ' 2026-03-20 16:26:25.709931 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.709935 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.709943 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.709947 | controller | - ' ' 2026-03-20 16:26:25.709951 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.709955 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.709959 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.709964 | controller | - ' ' 2026-03-20 16:26:25.709968 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.709972 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.709976 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.709981 | controller | available"' 2026-03-20 16:26:25.709985 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.709991 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.709995 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.709999 | controller | - ' exit 1' 2026-03-20 16:26:25.710003 | controller | - ' fi' 2026-03-20 16:26:25.710008 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.710012 | controller | - ' ' 2026-03-20 16:26:25.710016 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.710020 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.710024 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.710029 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.710033 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.710037 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.710041 | controller | -- \' 2026-03-20 16:26:25.710045 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.710052 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.710057 | controller | - ' logger.go:42: 16:09:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.710061 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.710065 | controller | - ' logger.go:42: 16:09:45 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.710069 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.710074 | controller | found' 2026-03-20 16:26:25.710078 | controller | - ' logger.go:42: 16:09:46 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.710082 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.710086 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.710091 | controller | - ' ' 2026-03-20 16:26:25.710097 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.710101 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.710105 | controller | - ' ' 2026-03-20 16:26:25.710109 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.710114 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.710118 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.710124 | controller | - ' exit 1' 2026-03-20 16:26:25.710128 | controller | - ' fi' 2026-03-20 16:26:25.710132 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.710136 | controller | - ' ' 2026-03-20 16:26:25.710141 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.710145 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.710149 | controller | | base64 -d)' 2026-03-20 16:26:25.710153 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.710158 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.710162 | controller | - ' exit 1' 2026-03-20 16:26:25.710166 | controller | - ' fi' 2026-03-20 16:26:25.710170 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.710174 | controller | - ' ' 2026-03-20 16:26:25.710179 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.710183 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.710194 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.710198 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.710202 | controller | --timeout=300s' 2026-03-20 16:26:25.710207 | controller | - ' ' 2026-03-20 16:26:25.710211 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.710215 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.710220 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.710224 | controller | - ' ' 2026-03-20 16:26:25.710228 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.710232 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.710236 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.710240 | controller | - ' ' 2026-03-20 16:26:25.710245 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.710249 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.710253 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.710257 | controller | available"' 2026-03-20 16:26:25.710261 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.710266 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.710270 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.710274 | controller | - ' exit 1' 2026-03-20 16:26:25.710280 | controller | - ' fi' 2026-03-20 16:26:25.710284 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.710288 | controller | - ' ' 2026-03-20 16:26:25.710293 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.710297 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.710301 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.710305 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.710309 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.710315 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.710319 | controller | -- \' 2026-03-20 16:26:25.710324 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.710328 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.710332 | controller | - ' logger.go:42: 16:09:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.710336 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.710340 | controller | - ' logger.go:42: 16:09:46 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.710344 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.710349 | controller | found' 2026-03-20 16:26:25.710353 | controller | - ' logger.go:42: 16:09:47 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.710357 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.710361 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.710365 | controller | - ' ' 2026-03-20 16:26:25.710370 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.710374 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.710378 | controller | - ' ' 2026-03-20 16:26:25.710382 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.710386 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.710390 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.710395 | controller | - ' exit 1' 2026-03-20 16:26:25.710399 | controller | - ' fi' 2026-03-20 16:26:25.710403 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.710407 | controller | - ' ' 2026-03-20 16:26:25.710411 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.710416 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.710420 | controller | | base64 -d)' 2026-03-20 16:26:25.710424 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.710428 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.710432 | controller | - ' exit 1' 2026-03-20 16:26:25.710437 | controller | - ' fi' 2026-03-20 16:26:25.710441 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.710445 | controller | - ' ' 2026-03-20 16:26:25.710449 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.710453 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.710457 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.710468 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.710473 | controller | --timeout=300s' 2026-03-20 16:26:25.710477 | controller | - ' ' 2026-03-20 16:26:25.710481 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.710486 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.710490 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.710494 | controller | - ' ' 2026-03-20 16:26:25.710498 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.710502 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.710508 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.710513 | controller | - ' ' 2026-03-20 16:26:25.710517 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.710521 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.710525 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.710529 | controller | available"' 2026-03-20 16:26:25.710533 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.710538 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.710542 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.710546 | controller | - ' exit 1' 2026-03-20 16:26:25.710550 | controller | - ' fi' 2026-03-20 16:26:25.710554 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.710558 | controller | - ' ' 2026-03-20 16:26:25.710562 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.710567 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.710571 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.710575 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.710579 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.710583 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.710587 | controller | -- \' 2026-03-20 16:26:25.710592 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.710596 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.710600 | controller | - ' logger.go:42: 16:09:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.710604 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.710617 | controller | - ' logger.go:42: 16:09:48 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.710622 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.710626 | controller | found' 2026-03-20 16:26:25.710630 | controller | - ' logger.go:42: 16:09:49 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.710634 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.710638 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.710643 | controller | - ' ' 2026-03-20 16:26:25.710647 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.710653 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.710657 | controller | - ' ' 2026-03-20 16:26:25.710661 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.710666 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.710670 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.710676 | controller | - ' exit 1' 2026-03-20 16:26:25.710680 | controller | - ' fi' 2026-03-20 16:26:25.710684 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.710688 | controller | - ' ' 2026-03-20 16:26:25.710707 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.710713 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.710717 | controller | | base64 -d)' 2026-03-20 16:26:25.710722 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.710728 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.710732 | controller | - ' exit 1' 2026-03-20 16:26:25.710736 | controller | - ' fi' 2026-03-20 16:26:25.710741 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.710745 | controller | - ' ' 2026-03-20 16:26:25.710749 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.710753 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.710758 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.710769 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.710774 | controller | --timeout=300s' 2026-03-20 16:26:25.710778 | controller | - ' ' 2026-03-20 16:26:25.710782 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.710787 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.710791 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.710795 | controller | - ' ' 2026-03-20 16:26:25.710799 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.710804 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.710808 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.710812 | controller | - ' ' 2026-03-20 16:26:25.710817 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.710821 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.710825 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.710829 | controller | available"' 2026-03-20 16:26:25.710834 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.710838 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.710842 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.710846 | controller | - ' exit 1' 2026-03-20 16:26:25.710851 | controller | - ' fi' 2026-03-20 16:26:25.710855 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.710859 | controller | - ' ' 2026-03-20 16:26:25.710863 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.710868 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.710872 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.710876 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.710882 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.710887 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.710891 | controller | -- \' 2026-03-20 16:26:25.710895 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.710900 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.710904 | controller | - ' logger.go:42: 16:09:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.710910 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.710914 | controller | - ' logger.go:42: 16:09:49 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.710918 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.710922 | controller | found' 2026-03-20 16:26:25.710927 | controller | - ' logger.go:42: 16:09:50 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.710931 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.710935 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.710940 | controller | - ' ' 2026-03-20 16:26:25.710944 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.710948 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.710952 | controller | - ' ' 2026-03-20 16:26:25.710957 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.710961 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.710965 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.710969 | controller | - ' exit 1' 2026-03-20 16:26:25.710974 | controller | - ' fi' 2026-03-20 16:26:25.710978 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.710982 | controller | - ' ' 2026-03-20 16:26:25.710986 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.710990 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.710995 | controller | | base64 -d)' 2026-03-20 16:26:25.710999 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.711003 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.711008 | controller | - ' exit 1' 2026-03-20 16:26:25.711012 | controller | - ' fi' 2026-03-20 16:26:25.711016 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.711020 | controller | - ' ' 2026-03-20 16:26:25.711025 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.711029 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.711033 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.711038 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.711056 | controller | --timeout=300s' 2026-03-20 16:26:25.711061 | controller | - ' ' 2026-03-20 16:26:25.711065 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.711069 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.711074 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.711078 | controller | - ' ' 2026-03-20 16:26:25.711082 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.711087 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.711091 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.711095 | controller | - ' ' 2026-03-20 16:26:25.711099 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.711104 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.711108 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.711112 | controller | available"' 2026-03-20 16:26:25.711116 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.711122 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.711127 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.711131 | controller | - ' exit 1' 2026-03-20 16:26:25.711135 | controller | - ' fi' 2026-03-20 16:26:25.711140 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.711144 | controller | - ' ' 2026-03-20 16:26:25.711148 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.711153 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.711157 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.711161 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.711165 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.711170 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.711174 | controller | -- \' 2026-03-20 16:26:25.711178 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.711182 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.711187 | controller | - ' logger.go:42: 16:09:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.711191 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.711195 | controller | - ' logger.go:42: 16:09:50 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.711199 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.711204 | controller | found' 2026-03-20 16:26:25.711208 | controller | - ' logger.go:42: 16:09:51 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.711212 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.711216 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.711221 | controller | - ' ' 2026-03-20 16:26:25.711225 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.711229 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.711234 | controller | - ' ' 2026-03-20 16:26:25.711240 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.711244 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.711248 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.711253 | controller | - ' exit 1' 2026-03-20 16:26:25.711257 | controller | - ' fi' 2026-03-20 16:26:25.711261 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.711266 | controller | - ' ' 2026-03-20 16:26:25.711271 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.711276 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.711280 | controller | | base64 -d)' 2026-03-20 16:26:25.711284 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.711289 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.711293 | controller | - ' exit 1' 2026-03-20 16:26:25.711297 | controller | - ' fi' 2026-03-20 16:26:25.711301 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.711306 | controller | - ' ' 2026-03-20 16:26:25.711310 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.711314 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.711320 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.711325 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.711329 | controller | --timeout=300s' 2026-03-20 16:26:25.711336 | controller | - ' ' 2026-03-20 16:26:25.711348 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.711352 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.711357 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.711361 | controller | - ' ' 2026-03-20 16:26:25.711365 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.711370 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.711374 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.711378 | controller | - ' ' 2026-03-20 16:26:25.711382 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.711387 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.711391 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.711395 | controller | available"' 2026-03-20 16:26:25.711400 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.711404 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.711408 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.711412 | controller | - ' exit 1' 2026-03-20 16:26:25.711417 | controller | - ' fi' 2026-03-20 16:26:25.711421 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.711425 | controller | - ' ' 2026-03-20 16:26:25.711429 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.711434 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.711438 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.711442 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.711447 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.711451 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.711455 | controller | -- \' 2026-03-20 16:26:25.711459 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.711464 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.711468 | controller | - ' logger.go:42: 16:09:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.711472 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.711477 | controller | - ' logger.go:42: 16:09:51 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.711481 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.711486 | controller | found' 2026-03-20 16:26:25.711492 | controller | - ' logger.go:42: 16:09:52 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.711498 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.711504 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.711510 | controller | - ' ' 2026-03-20 16:26:25.711518 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.711524 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.711530 | controller | - ' ' 2026-03-20 16:26:25.711534 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.711539 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.711543 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.711547 | controller | - ' exit 1' 2026-03-20 16:26:25.711551 | controller | - ' fi' 2026-03-20 16:26:25.711556 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.711560 | controller | - ' ' 2026-03-20 16:26:25.711566 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.711572 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.711577 | controller | | base64 -d)' 2026-03-20 16:26:25.711583 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.711589 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.711594 | controller | - ' exit 1' 2026-03-20 16:26:25.711598 | controller | - ' fi' 2026-03-20 16:26:25.711602 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.711607 | controller | - ' ' 2026-03-20 16:26:25.711647 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.711652 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.711656 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.711660 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.711665 | controller | --timeout=300s' 2026-03-20 16:26:25.711669 | controller | - ' ' 2026-03-20 16:26:25.711682 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.711686 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.711691 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.711710 | controller | - ' ' 2026-03-20 16:26:25.711715 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.711719 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.711723 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.711729 | controller | - ' ' 2026-03-20 16:26:25.711734 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.711739 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.711745 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.711751 | controller | available"' 2026-03-20 16:26:25.711756 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.711762 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.711768 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.711774 | controller | - ' exit 1' 2026-03-20 16:26:25.711779 | controller | - ' fi' 2026-03-20 16:26:25.711785 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.711791 | controller | - ' ' 2026-03-20 16:26:25.711796 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.711805 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.711811 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.711815 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.711820 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.711824 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.711828 | controller | -- \' 2026-03-20 16:26:25.711833 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.711837 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.711841 | controller | - ' logger.go:42: 16:09:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.711845 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.711852 | controller | - ' logger.go:42: 16:09:52 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.711857 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.711861 | controller | found' 2026-03-20 16:26:25.711865 | controller | - ' logger.go:42: 16:09:53 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.711870 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.711874 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.711878 | controller | - ' ' 2026-03-20 16:26:25.711883 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.711887 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.711891 | controller | - ' ' 2026-03-20 16:26:25.711895 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.711899 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.711904 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.711908 | controller | - ' exit 1' 2026-03-20 16:26:25.711912 | controller | - ' fi' 2026-03-20 16:26:25.711916 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.711922 | controller | - ' ' 2026-03-20 16:26:25.711926 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.711931 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.711936 | controller | | base64 -d)' 2026-03-20 16:26:25.711941 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.711947 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.711953 | controller | - ' exit 1' 2026-03-20 16:26:25.711959 | controller | - ' fi' 2026-03-20 16:26:25.711964 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.711970 | controller | - ' ' 2026-03-20 16:26:25.711976 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.711980 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.711984 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.711989 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.711993 | controller | --timeout=300s' 2026-03-20 16:26:25.711997 | controller | - ' ' 2026-03-20 16:26:25.712001 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.712015 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.712022 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.712027 | controller | - ' ' 2026-03-20 16:26:25.712031 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.712037 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.712042 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.712046 | controller | - ' ' 2026-03-20 16:26:25.712050 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.712054 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.712060 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.712064 | controller | available"' 2026-03-20 16:26:25.712068 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.712073 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.712077 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.712081 | controller | - ' exit 1' 2026-03-20 16:26:25.712085 | controller | - ' fi' 2026-03-20 16:26:25.712090 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.712094 | controller | - ' ' 2026-03-20 16:26:25.712098 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.712102 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.712106 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.712111 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.712115 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.712119 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.712123 | controller | -- \' 2026-03-20 16:26:25.712127 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.712132 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.712138 | controller | - ' logger.go:42: 16:09:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.712143 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.712149 | controller | - ' logger.go:42: 16:09:53 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.712155 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.712161 | controller | found' 2026-03-20 16:26:25.712166 | controller | - ' logger.go:42: 16:09:54 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.712172 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.712178 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.712183 | controller | - ' ' 2026-03-20 16:26:25.712188 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.712192 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.712196 | controller | - ' ' 2026-03-20 16:26:25.712200 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.712204 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.712209 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.712215 | controller | - ' exit 1' 2026-03-20 16:26:25.712219 | controller | - ' fi' 2026-03-20 16:26:25.712224 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.712228 | controller | - ' ' 2026-03-20 16:26:25.712232 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.712236 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.712240 | controller | | base64 -d)' 2026-03-20 16:26:25.712245 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.712249 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.712253 | controller | - ' exit 1' 2026-03-20 16:26:25.712257 | controller | - ' fi' 2026-03-20 16:26:25.712262 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.712266 | controller | - ' ' 2026-03-20 16:26:25.712270 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.712274 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.712278 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.712283 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.712287 | controller | --timeout=300s' 2026-03-20 16:26:25.712291 | controller | - ' ' 2026-03-20 16:26:25.712296 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.712312 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.712319 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.712324 | controller | - ' ' 2026-03-20 16:26:25.712330 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.712336 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.712341 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.712346 | controller | - ' ' 2026-03-20 16:26:25.712351 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.712357 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.712362 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.712366 | controller | available"' 2026-03-20 16:26:25.712371 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.712375 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.712379 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.712383 | controller | - ' exit 1' 2026-03-20 16:26:25.712388 | controller | - ' fi' 2026-03-20 16:26:25.712392 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.712396 | controller | - ' ' 2026-03-20 16:26:25.712400 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.712404 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.712410 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.712415 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.712421 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.712426 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.712435 | controller | -- \' 2026-03-20 16:26:25.712439 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.712443 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.712447 | controller | - ' logger.go:42: 16:09:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.712452 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.712456 | controller | - ' logger.go:42: 16:09:55 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.712460 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.712464 | controller | found' 2026-03-20 16:26:25.712468 | controller | - ' logger.go:42: 16:09:56 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.712473 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.712477 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.712481 | controller | - ' ' 2026-03-20 16:26:25.712485 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.712489 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.712494 | controller | - ' ' 2026-03-20 16:26:25.712498 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.712502 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.712506 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.712513 | controller | - ' exit 1' 2026-03-20 16:26:25.712517 | controller | - ' fi' 2026-03-20 16:26:25.712522 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.712526 | controller | - ' ' 2026-03-20 16:26:25.712532 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.712536 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.712542 | controller | | base64 -d)' 2026-03-20 16:26:25.712547 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.712553 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.712559 | controller | - ' exit 1' 2026-03-20 16:26:25.712565 | controller | - ' fi' 2026-03-20 16:26:25.712570 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.712576 | controller | - ' ' 2026-03-20 16:26:25.712582 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.712586 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.712593 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.712597 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.712601 | controller | --timeout=300s' 2026-03-20 16:26:25.712605 | controller | - ' ' 2026-03-20 16:26:25.712610 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.712614 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.712628 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.712632 | controller | - ' ' 2026-03-20 16:26:25.712637 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.712642 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.712648 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.712657 | controller | - ' ' 2026-03-20 16:26:25.712661 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.712666 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.712670 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.712674 | controller | available"' 2026-03-20 16:26:25.712678 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.712686 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.712709 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.712719 | controller | - ' exit 1' 2026-03-20 16:26:25.712724 | controller | - ' fi' 2026-03-20 16:26:25.712729 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.712735 | controller | - ' ' 2026-03-20 16:26:25.712741 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.712746 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.712752 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.712757 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.712761 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.712765 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.712770 | controller | -- \' 2026-03-20 16:26:25.712774 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.712778 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.712782 | controller | - ' logger.go:42: 16:09:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.712786 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.712791 | controller | - ' logger.go:42: 16:09:56 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.712795 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.712799 | controller | found' 2026-03-20 16:26:25.712803 | controller | - ' logger.go:42: 16:09:57 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.712807 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.712812 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.712816 | controller | - ' ' 2026-03-20 16:26:25.712820 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.712826 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.712831 | controller | - ' ' 2026-03-20 16:26:25.712837 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.712842 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.712848 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.712854 | controller | - ' exit 1' 2026-03-20 16:26:25.712860 | controller | - ' fi' 2026-03-20 16:26:25.712866 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.712871 | controller | - ' ' 2026-03-20 16:26:25.712877 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.712881 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.712886 | controller | | base64 -d)' 2026-03-20 16:26:25.712890 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.712898 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.712902 | controller | - ' exit 1' 2026-03-20 16:26:25.712906 | controller | - ' fi' 2026-03-20 16:26:25.712911 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.712915 | controller | - ' ' 2026-03-20 16:26:25.712919 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.712923 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.712928 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.712932 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.712936 | controller | --timeout=300s' 2026-03-20 16:26:25.712940 | controller | - ' ' 2026-03-20 16:26:25.712944 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.712949 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.712962 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.712967 | controller | - ' ' 2026-03-20 16:26:25.712971 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.712976 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.712980 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.712986 | controller | - ' ' 2026-03-20 16:26:25.712991 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.712997 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.713003 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.713007 | controller | available"' 2026-03-20 16:26:25.713012 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.713016 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.713020 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.713024 | controller | - ' exit 1' 2026-03-20 16:26:25.713028 | controller | - ' fi' 2026-03-20 16:26:25.713033 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.713037 | controller | - ' ' 2026-03-20 16:26:25.713041 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.713045 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.713050 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.713054 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.713058 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.713062 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.713069 | controller | -- \' 2026-03-20 16:26:25.713074 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.713078 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.713082 | controller | - ' logger.go:42: 16:09:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.713086 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.713093 | controller | - ' logger.go:42: 16:09:57 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.713097 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.713101 | controller | found' 2026-03-20 16:26:25.713105 | controller | - ' logger.go:42: 16:09:58 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.713110 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.713114 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.713118 | controller | - ' ' 2026-03-20 16:26:25.713122 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.713126 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.713131 | controller | - ' ' 2026-03-20 16:26:25.713135 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.713139 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.713143 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.713147 | controller | - ' exit 1' 2026-03-20 16:26:25.713152 | controller | - ' fi' 2026-03-20 16:26:25.713156 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.713160 | controller | - ' ' 2026-03-20 16:26:25.713164 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.713168 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.713173 | controller | | base64 -d)' 2026-03-20 16:26:25.713177 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.713181 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.713185 | controller | - ' exit 1' 2026-03-20 16:26:25.713189 | controller | - ' fi' 2026-03-20 16:26:25.713193 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.713198 | controller | - ' ' 2026-03-20 16:26:25.713202 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.713206 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.713210 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.713216 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.713221 | controller | --timeout=300s' 2026-03-20 16:26:25.713225 | controller | - ' ' 2026-03-20 16:26:25.713229 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.713233 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.713238 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.713250 | controller | - ' ' 2026-03-20 16:26:25.713255 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.713259 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.713264 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.713268 | controller | - ' ' 2026-03-20 16:26:25.713272 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.713276 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.713280 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.713284 | controller | available"' 2026-03-20 16:26:25.713289 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.713295 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.713299 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.713303 | controller | - ' exit 1' 2026-03-20 16:26:25.713307 | controller | - ' fi' 2026-03-20 16:26:25.713311 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.713316 | controller | - ' ' 2026-03-20 16:26:25.713320 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.713324 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.713328 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.713332 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.713337 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.713341 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.713345 | controller | -- \' 2026-03-20 16:26:25.713349 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.713353 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.713357 | controller | - ' logger.go:42: 16:09:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.713362 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.713366 | controller | - ' logger.go:42: 16:09:58 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.713370 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.713374 | controller | found' 2026-03-20 16:26:25.713378 | controller | - ' logger.go:42: 16:09:59 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.713382 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.713387 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.713391 | controller | - ' ' 2026-03-20 16:26:25.713395 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.713399 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.713403 | controller | - ' ' 2026-03-20 16:26:25.713408 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.713412 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.713416 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.713420 | controller | - ' exit 1' 2026-03-20 16:26:25.713424 | controller | - ' fi' 2026-03-20 16:26:25.713429 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.713433 | controller | - ' ' 2026-03-20 16:26:25.713437 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.713441 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.713445 | controller | | base64 -d)' 2026-03-20 16:26:25.713450 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.713454 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.713458 | controller | - ' exit 1' 2026-03-20 16:26:25.713462 | controller | - ' fi' 2026-03-20 16:26:25.713466 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.713472 | controller | - ' ' 2026-03-20 16:26:25.713477 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.713481 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.713485 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.713491 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.713495 | controller | --timeout=300s' 2026-03-20 16:26:25.713499 | controller | - ' ' 2026-03-20 16:26:25.713504 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.713508 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.713512 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.713516 | controller | - ' ' 2026-03-20 16:26:25.713527 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.713532 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.713536 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.713540 | controller | - ' ' 2026-03-20 16:26:25.713544 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.713548 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.713553 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.713557 | controller | available"' 2026-03-20 16:26:25.713561 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.713565 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.713569 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.713574 | controller | - ' exit 1' 2026-03-20 16:26:25.713578 | controller | - ' fi' 2026-03-20 16:26:25.713582 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.713586 | controller | - ' ' 2026-03-20 16:26:25.713593 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.713597 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.713601 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.713605 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.713620 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.713624 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.713629 | controller | -- \' 2026-03-20 16:26:25.713633 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.713637 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.713641 | controller | - ' logger.go:42: 16:09:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.713646 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.713650 | controller | - ' logger.go:42: 16:09:59 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.713654 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.713658 | controller | found' 2026-03-20 16:26:25.713663 | controller | - ' logger.go:42: 16:10:00 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.713667 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.713671 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.713675 | controller | - ' ' 2026-03-20 16:26:25.713679 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.713686 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.713709 | controller | - ' ' 2026-03-20 16:26:25.713716 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.713720 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.713725 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.713729 | controller | - ' exit 1' 2026-03-20 16:26:25.713733 | controller | - ' fi' 2026-03-20 16:26:25.713737 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.713742 | controller | - ' ' 2026-03-20 16:26:25.713746 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.713750 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.713754 | controller | | base64 -d)' 2026-03-20 16:26:25.713758 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.713763 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.713767 | controller | - ' exit 1' 2026-03-20 16:26:25.713771 | controller | - ' fi' 2026-03-20 16:26:25.713775 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.713780 | controller | - ' ' 2026-03-20 16:26:25.713792 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.713796 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.713800 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.713804 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.713808 | controller | --timeout=300s' 2026-03-20 16:26:25.713815 | controller | - ' ' 2026-03-20 16:26:25.713819 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.713823 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.713828 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.713832 | controller | - ' ' 2026-03-20 16:26:25.713843 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.713848 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.713852 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.713856 | controller | - ' ' 2026-03-20 16:26:25.713868 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.713872 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.713876 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.713881 | controller | available"' 2026-03-20 16:26:25.713885 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.713889 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.713893 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.713897 | controller | - ' exit 1' 2026-03-20 16:26:25.713904 | controller | - ' fi' 2026-03-20 16:26:25.713908 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.713912 | controller | - ' ' 2026-03-20 16:26:25.713916 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.713920 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.713927 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.713931 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.713935 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.713940 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.713944 | controller | -- \' 2026-03-20 16:26:25.713948 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.713952 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.713956 | controller | - ' logger.go:42: 16:10:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.713968 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.713973 | controller | - ' logger.go:42: 16:10:00 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.713977 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.713981 | controller | found' 2026-03-20 16:26:25.713985 | controller | - ' logger.go:42: 16:10:01 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.713989 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.713994 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.713998 | controller | - ' ' 2026-03-20 16:26:25.714002 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.714006 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.714010 | controller | - ' ' 2026-03-20 16:26:25.714015 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.714019 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.714023 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.714027 | controller | - ' exit 1' 2026-03-20 16:26:25.714032 | controller | - ' fi' 2026-03-20 16:26:25.714036 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.714040 | controller | - ' ' 2026-03-20 16:26:25.714046 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.714050 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.714054 | controller | | base64 -d)' 2026-03-20 16:26:25.714059 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.714063 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.714067 | controller | - ' exit 1' 2026-03-20 16:26:25.714071 | controller | - ' fi' 2026-03-20 16:26:25.714075 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.714080 | controller | - ' ' 2026-03-20 16:26:25.714084 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.714088 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.714092 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.714096 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.714100 | controller | --timeout=300s' 2026-03-20 16:26:25.714105 | controller | - ' ' 2026-03-20 16:26:25.714109 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.714113 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.714119 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.714124 | controller | - ' ' 2026-03-20 16:26:25.714128 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.714140 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.714146 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.714150 | controller | - ' ' 2026-03-20 16:26:25.714155 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.714159 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.714163 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.714167 | controller | available"' 2026-03-20 16:26:25.714171 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.714176 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.714180 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.714184 | controller | - ' exit 1' 2026-03-20 16:26:25.714188 | controller | - ' fi' 2026-03-20 16:26:25.714192 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.714197 | controller | - ' ' 2026-03-20 16:26:25.714201 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.714205 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.714209 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.714213 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.714217 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.714222 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.714226 | controller | -- \' 2026-03-20 16:26:25.714230 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.714234 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.714238 | controller | - ' logger.go:42: 16:10:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.714243 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.714247 | controller | - ' logger.go:42: 16:10:02 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.714251 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.714255 | controller | found' 2026-03-20 16:26:25.714260 | controller | - ' logger.go:42: 16:10:03 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.714264 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.714268 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.714272 | controller | - ' ' 2026-03-20 16:26:25.714276 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.714281 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.714285 | controller | - ' ' 2026-03-20 16:26:25.714289 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.714293 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.714297 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.714302 | controller | - ' exit 1' 2026-03-20 16:26:25.714309 | controller | - ' fi' 2026-03-20 16:26:25.714314 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.714318 | controller | - ' ' 2026-03-20 16:26:25.714322 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.714326 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.714330 | controller | | base64 -d)' 2026-03-20 16:26:25.714335 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.714339 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.714343 | controller | - ' exit 1' 2026-03-20 16:26:25.714347 | controller | - ' fi' 2026-03-20 16:26:25.714351 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.714355 | controller | - ' ' 2026-03-20 16:26:25.714360 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.714364 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.714368 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.714372 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.714376 | controller | --timeout=300s' 2026-03-20 16:26:25.714380 | controller | - ' ' 2026-03-20 16:26:25.714385 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.714389 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.714393 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.714397 | controller | - ' ' 2026-03-20 16:26:25.714401 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.714414 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.714419 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.714423 | controller | - ' ' 2026-03-20 16:26:25.714428 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.714432 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.714436 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.714440 | controller | available"' 2026-03-20 16:26:25.714444 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.714449 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.714453 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.714457 | controller | - ' exit 1' 2026-03-20 16:26:25.714461 | controller | - ' fi' 2026-03-20 16:26:25.714465 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.714469 | controller | - ' ' 2026-03-20 16:26:25.714474 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.714478 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.714482 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.714486 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.714490 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.714494 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.714509 | controller | -- \' 2026-03-20 16:26:25.714513 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.714517 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.714522 | controller | - ' logger.go:42: 16:10:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.714526 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.714530 | controller | - ' logger.go:42: 16:10:03 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.714534 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.714538 | controller | found' 2026-03-20 16:26:25.714544 | controller | - ' logger.go:42: 16:10:04 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.714548 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.714553 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.714557 | controller | - ' ' 2026-03-20 16:26:25.714561 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.714565 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.714569 | controller | - ' ' 2026-03-20 16:26:25.714574 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.714578 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.714582 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.714586 | controller | - ' exit 1' 2026-03-20 16:26:25.714590 | controller | - ' fi' 2026-03-20 16:26:25.714595 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.714599 | controller | - ' ' 2026-03-20 16:26:25.714603 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.714607 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.714612 | controller | | base64 -d)' 2026-03-20 16:26:25.714616 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.714620 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.714624 | controller | - ' exit 1' 2026-03-20 16:26:25.714628 | controller | - ' fi' 2026-03-20 16:26:25.714633 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.714637 | controller | - ' ' 2026-03-20 16:26:25.714641 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.714645 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.714649 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.714654 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.714658 | controller | --timeout=300s' 2026-03-20 16:26:25.714662 | controller | - ' ' 2026-03-20 16:26:25.714666 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.714670 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.714675 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.714679 | controller | - ' ' 2026-03-20 16:26:25.714685 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.714689 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.714713 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.714720 | controller | - ' ' 2026-03-20 16:26:25.714724 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.714728 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.714733 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.714738 | controller | available"' 2026-03-20 16:26:25.714752 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.714758 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.714763 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.714768 | controller | - ' exit 1' 2026-03-20 16:26:25.714774 | controller | - ' fi' 2026-03-20 16:26:25.714780 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.714786 | controller | - ' ' 2026-03-20 16:26:25.714791 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.714797 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.714802 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.714807 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.714812 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.714817 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.714823 | controller | -- \' 2026-03-20 16:26:25.714828 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.714833 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.714839 | controller | - ' logger.go:42: 16:10:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.714844 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.714849 | controller | - ' logger.go:42: 16:10:04 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.714854 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.714860 | controller | found' 2026-03-20 16:26:25.714866 | controller | - ' logger.go:42: 16:10:05 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.714871 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.714877 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.714882 | controller | - ' ' 2026-03-20 16:26:25.714888 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.714893 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.714899 | controller | - ' ' 2026-03-20 16:26:25.714904 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.714909 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.714913 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.714921 | controller | - ' exit 1' 2026-03-20 16:26:25.714925 | controller | - ' fi' 2026-03-20 16:26:25.714929 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.714933 | controller | - ' ' 2026-03-20 16:26:25.714937 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.714942 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.714946 | controller | | base64 -d)' 2026-03-20 16:26:25.714951 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.714956 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.714964 | controller | - ' exit 1' 2026-03-20 16:26:25.714970 | controller | - ' fi' 2026-03-20 16:26:25.714976 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.714981 | controller | - ' ' 2026-03-20 16:26:25.714987 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.714991 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.714995 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.715001 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.715006 | controller | --timeout=300s' 2026-03-20 16:26:25.715010 | controller | - ' ' 2026-03-20 16:26:25.715014 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.715018 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.715022 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.715027 | controller | - ' ' 2026-03-20 16:26:25.715031 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.715035 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.715049 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.715054 | controller | - ' ' 2026-03-20 16:26:25.715058 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.715062 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.715066 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.715071 | controller | available"' 2026-03-20 16:26:25.715075 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.715081 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.715085 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.715090 | controller | - ' exit 1' 2026-03-20 16:26:25.715095 | controller | - ' fi' 2026-03-20 16:26:25.715101 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.715107 | controller | - ' ' 2026-03-20 16:26:25.715112 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.715118 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.715123 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.715127 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.715132 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.715136 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.715140 | controller | -- \' 2026-03-20 16:26:25.715144 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.715148 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.715155 | controller | - ' logger.go:42: 16:10:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.715159 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.715163 | controller | - ' logger.go:42: 16:10:05 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.715171 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.715175 | controller | found' 2026-03-20 16:26:25.715180 | controller | - ' logger.go:42: 16:10:06 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.715184 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.715188 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.715192 | controller | - ' ' 2026-03-20 16:26:25.715196 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.715201 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.715205 | controller | - ' ' 2026-03-20 16:26:25.715209 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.715213 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.715217 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.715222 | controller | - ' exit 1' 2026-03-20 16:26:25.715226 | controller | - ' fi' 2026-03-20 16:26:25.715230 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.715234 | controller | - ' ' 2026-03-20 16:26:25.715238 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.715242 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.715247 | controller | | base64 -d)' 2026-03-20 16:26:25.715251 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.715255 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.715259 | controller | - ' exit 1' 2026-03-20 16:26:25.715263 | controller | - ' fi' 2026-03-20 16:26:25.715268 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.715274 | controller | - ' ' 2026-03-20 16:26:25.715279 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.715285 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.715290 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.715296 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.715302 | controller | --timeout=300s' 2026-03-20 16:26:25.715306 | controller | - ' ' 2026-03-20 16:26:25.715310 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.715315 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.715319 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.715323 | controller | - ' ' 2026-03-20 16:26:25.715328 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.715334 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.715340 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.715345 | controller | - ' ' 2026-03-20 16:26:25.715361 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.715365 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.715371 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.715376 | controller | available"' 2026-03-20 16:26:25.715382 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.715391 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.715400 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.715405 | controller | - ' exit 1' 2026-03-20 16:26:25.715411 | controller | - ' fi' 2026-03-20 16:26:25.715416 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.715422 | controller | - ' ' 2026-03-20 16:26:25.715428 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.715433 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.715437 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.715442 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.715446 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.715450 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.715454 | controller | -- \' 2026-03-20 16:26:25.715459 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.715463 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.715467 | controller | - ' logger.go:42: 16:10:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.715472 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.715477 | controller | - ' logger.go:42: 16:10:06 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.715483 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.715489 | controller | found' 2026-03-20 16:26:25.715494 | controller | - ' logger.go:42: 16:10:07 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.715500 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.715506 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.715515 | controller | - ' ' 2026-03-20 16:26:25.715520 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.715524 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.715529 | controller | - ' ' 2026-03-20 16:26:25.715533 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.715537 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.715542 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.715547 | controller | - ' exit 1' 2026-03-20 16:26:25.715553 | controller | - ' fi' 2026-03-20 16:26:25.715559 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.715564 | controller | - ' ' 2026-03-20 16:26:25.715570 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.715574 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.715579 | controller | | base64 -d)' 2026-03-20 16:26:25.715583 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.715587 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.715591 | controller | - ' exit 1' 2026-03-20 16:26:25.715596 | controller | - ' fi' 2026-03-20 16:26:25.715600 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.715604 | controller | - ' ' 2026-03-20 16:26:25.715608 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.715613 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.715651 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.715658 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.715663 | controller | --timeout=300s' 2026-03-20 16:26:25.715667 | controller | - ' ' 2026-03-20 16:26:25.715672 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.715676 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.715680 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.715684 | controller | - ' ' 2026-03-20 16:26:25.715689 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.715705 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.715712 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.715717 | controller | - ' ' 2026-03-20 16:26:25.715731 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.715735 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.715740 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.715744 | controller | available"' 2026-03-20 16:26:25.715749 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.715754 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.715758 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.715762 | controller | - ' exit 1' 2026-03-20 16:26:25.715766 | controller | - ' fi' 2026-03-20 16:26:25.715770 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.715775 | controller | - ' ' 2026-03-20 16:26:25.715783 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.715787 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.715791 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.715797 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.715803 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.715809 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.715815 | controller | -- \' 2026-03-20 16:26:25.715820 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.715825 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.715829 | controller | - ' logger.go:42: 16:10:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.715834 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.715838 | controller | - ' logger.go:42: 16:10:07 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.715842 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.715846 | controller | found' 2026-03-20 16:26:25.715851 | controller | - ' logger.go:42: 16:10:08 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.715855 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.715859 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.715863 | controller | - ' ' 2026-03-20 16:26:25.715868 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.715874 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.715879 | controller | - ' ' 2026-03-20 16:26:25.715883 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.715887 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.715891 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.715895 | controller | - ' exit 1' 2026-03-20 16:26:25.715900 | controller | - ' fi' 2026-03-20 16:26:25.715904 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.715908 | controller | - ' ' 2026-03-20 16:26:25.715912 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.715917 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.715921 | controller | | base64 -d)' 2026-03-20 16:26:25.715925 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.715929 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.715935 | controller | - ' exit 1' 2026-03-20 16:26:25.715944 | controller | - ' fi' 2026-03-20 16:26:25.715950 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.715954 | controller | - ' ' 2026-03-20 16:26:25.715958 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.715962 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.715967 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.715971 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.715976 | controller | --timeout=300s' 2026-03-20 16:26:25.715982 | controller | - ' ' 2026-03-20 16:26:25.715988 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.715994 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.715999 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.716003 | controller | - ' ' 2026-03-20 16:26:25.716008 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.716012 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.716016 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.716020 | controller | - ' ' 2026-03-20 16:26:25.716025 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.716037 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.716042 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.716046 | controller | available"' 2026-03-20 16:26:25.716051 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.716055 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.716059 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.716063 | controller | - ' exit 1' 2026-03-20 16:26:25.716068 | controller | - ' fi' 2026-03-20 16:26:25.716072 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.716076 | controller | - ' ' 2026-03-20 16:26:25.716080 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.716084 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.716091 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.716095 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.716099 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.716103 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.716108 | controller | -- \' 2026-03-20 16:26:25.716113 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.716118 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.716124 | controller | - ' logger.go:42: 16:10:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.716129 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.716135 | controller | - ' logger.go:42: 16:10:08 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.716141 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.716147 | controller | found' 2026-03-20 16:26:25.716151 | controller | - ' logger.go:42: 16:10:09 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.716155 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.716159 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.716164 | controller | - ' ' 2026-03-20 16:26:25.716168 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.716172 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.716176 | controller | - ' ' 2026-03-20 16:26:25.716180 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.716185 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.716189 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.716193 | controller | - ' exit 1' 2026-03-20 16:26:25.716197 | controller | - ' fi' 2026-03-20 16:26:25.716202 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.716206 | controller | - ' ' 2026-03-20 16:26:25.716210 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.716214 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.716219 | controller | | base64 -d)' 2026-03-20 16:26:25.716223 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.716227 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.716231 | controller | - ' exit 1' 2026-03-20 16:26:25.716236 | controller | - ' fi' 2026-03-20 16:26:25.716240 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.716244 | controller | - ' ' 2026-03-20 16:26:25.716248 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.716252 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.716257 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.716261 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.716265 | controller | --timeout=300s' 2026-03-20 16:26:25.716269 | controller | - ' ' 2026-03-20 16:26:25.716274 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.716278 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.716282 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.716289 | controller | - ' ' 2026-03-20 16:26:25.716293 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.716298 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.716302 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.716306 | controller | - ' ' 2026-03-20 16:26:25.716310 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.716317 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.716330 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.716335 | controller | available"' 2026-03-20 16:26:25.716339 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.716344 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.716348 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.716352 | controller | - ' exit 1' 2026-03-20 16:26:25.716357 | controller | - ' fi' 2026-03-20 16:26:25.716361 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.716365 | controller | - ' ' 2026-03-20 16:26:25.716369 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.716374 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.716378 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.716388 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.716392 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.716398 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.716402 | controller | -- \' 2026-03-20 16:26:25.716406 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.716411 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.716415 | controller | - ' logger.go:42: 16:10:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.716419 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.716423 | controller | - ' logger.go:42: 16:10:10 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.716427 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.716432 | controller | found' 2026-03-20 16:26:25.716436 | controller | - ' logger.go:42: 16:10:11 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.716440 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.716444 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.716449 | controller | - ' ' 2026-03-20 16:26:25.716453 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.716457 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.716461 | controller | - ' ' 2026-03-20 16:26:25.716465 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.716470 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.716474 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.716478 | controller | - ' exit 1' 2026-03-20 16:26:25.716482 | controller | - ' fi' 2026-03-20 16:26:25.716487 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.716493 | controller | - ' ' 2026-03-20 16:26:25.716498 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.716502 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.716506 | controller | | base64 -d)' 2026-03-20 16:26:25.716511 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.716515 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.716519 | controller | - ' exit 1' 2026-03-20 16:26:25.716523 | controller | - ' fi' 2026-03-20 16:26:25.716527 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.716532 | controller | - ' ' 2026-03-20 16:26:25.716536 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.716540 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.716544 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.716548 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.716553 | controller | --timeout=300s' 2026-03-20 16:26:25.716557 | controller | - ' ' 2026-03-20 16:26:25.716561 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.716565 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.716569 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.716574 | controller | - ' ' 2026-03-20 16:26:25.716578 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.716582 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.716586 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.716590 | controller | - ' ' 2026-03-20 16:26:25.716594 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.716599 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.716610 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.716614 | controller | available"' 2026-03-20 16:26:25.716619 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.716623 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.716627 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.716631 | controller | - ' exit 1' 2026-03-20 16:26:25.716636 | controller | - ' fi' 2026-03-20 16:26:25.716642 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.716647 | controller | - ' ' 2026-03-20 16:26:25.716653 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.716658 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.716663 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.716668 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.716672 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.716676 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.716680 | controller | -- \' 2026-03-20 16:26:25.716687 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.716704 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.716711 | controller | - ' logger.go:42: 16:10:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.716715 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.716720 | controller | - ' logger.go:42: 16:10:11 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.716724 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.716728 | controller | found' 2026-03-20 16:26:25.716732 | controller | - ' logger.go:42: 16:10:12 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.716737 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.716741 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.716745 | controller | - ' ' 2026-03-20 16:26:25.716749 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.716753 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.716757 | controller | - ' ' 2026-03-20 16:26:25.716762 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.716766 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.716770 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.716774 | controller | - ' exit 1' 2026-03-20 16:26:25.716779 | controller | - ' fi' 2026-03-20 16:26:25.716783 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.716788 | controller | - ' ' 2026-03-20 16:26:25.716796 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.716802 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.716807 | controller | | base64 -d)' 2026-03-20 16:26:25.716813 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.716819 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.716826 | controller | - ' exit 1' 2026-03-20 16:26:25.716831 | controller | - ' fi' 2026-03-20 16:26:25.716837 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.716842 | controller | - ' ' 2026-03-20 16:26:25.716848 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.716853 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.716858 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.716862 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.716866 | controller | --timeout=300s' 2026-03-20 16:26:25.716870 | controller | - ' ' 2026-03-20 16:26:25.716874 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.716879 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.716883 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.716887 | controller | - ' ' 2026-03-20 16:26:25.716891 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.716896 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.716900 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.716904 | controller | - ' ' 2026-03-20 16:26:25.716908 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.716915 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.716919 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.716932 | controller | available"' 2026-03-20 16:26:25.716937 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.716941 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.716945 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.716949 | controller | - ' exit 1' 2026-03-20 16:26:25.716954 | controller | - ' fi' 2026-03-20 16:26:25.716958 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.716962 | controller | - ' ' 2026-03-20 16:26:25.716966 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.716970 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.716975 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.716979 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.716983 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.716987 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.716991 | controller | -- \' 2026-03-20 16:26:25.716996 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.717002 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.717006 | controller | - ' logger.go:42: 16:10:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.717011 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.717015 | controller | - ' logger.go:42: 16:10:12 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.717019 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.717023 | controller | found' 2026-03-20 16:26:25.717028 | controller | - ' logger.go:42: 16:10:13 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.717032 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.717036 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.717040 | controller | - ' ' 2026-03-20 16:26:25.717045 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.717049 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.717053 | controller | - ' ' 2026-03-20 16:26:25.717059 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.717063 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.717068 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.717072 | controller | - ' exit 1' 2026-03-20 16:26:25.717076 | controller | - ' fi' 2026-03-20 16:26:25.717081 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.717085 | controller | - ' ' 2026-03-20 16:26:25.717089 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.717093 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.717097 | controller | | base64 -d)' 2026-03-20 16:26:25.717102 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.717106 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.717110 | controller | - ' exit 1' 2026-03-20 16:26:25.717116 | controller | - ' fi' 2026-03-20 16:26:25.717120 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.717125 | controller | - ' ' 2026-03-20 16:26:25.717129 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.717133 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.717137 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.717141 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.717145 | controller | --timeout=300s' 2026-03-20 16:26:25.717150 | controller | - ' ' 2026-03-20 16:26:25.717154 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.717158 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.717162 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.717166 | controller | - ' ' 2026-03-20 16:26:25.717171 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.717175 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.717179 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.717183 | controller | - ' ' 2026-03-20 16:26:25.717187 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.717192 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.717196 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.717200 | controller | available"' 2026-03-20 16:26:25.717211 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.717215 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.717220 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.717224 | controller | - ' exit 1' 2026-03-20 16:26:25.717228 | controller | - ' fi' 2026-03-20 16:26:25.717232 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.717237 | controller | - ' ' 2026-03-20 16:26:25.717241 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.717245 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.717250 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.717256 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.717261 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.717268 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.717273 | controller | -- \' 2026-03-20 16:26:25.717279 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.717285 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.717290 | controller | - ' logger.go:42: 16:10:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.717294 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.717298 | controller | - ' logger.go:42: 16:10:13 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.717302 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.717310 | controller | found' 2026-03-20 16:26:25.717315 | controller | - ' logger.go:42: 16:10:14 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.717319 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.717323 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.717327 | controller | - ' ' 2026-03-20 16:26:25.717332 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.717336 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.717340 | controller | - ' ' 2026-03-20 16:26:25.717344 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.717349 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.717353 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.717357 | controller | - ' exit 1' 2026-03-20 16:26:25.717361 | controller | - ' fi' 2026-03-20 16:26:25.717365 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.717370 | controller | - ' ' 2026-03-20 16:26:25.717374 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.717378 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.717382 | controller | | base64 -d)' 2026-03-20 16:26:25.717386 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.717391 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.717395 | controller | - ' exit 1' 2026-03-20 16:26:25.717399 | controller | - ' fi' 2026-03-20 16:26:25.717403 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.717408 | controller | - ' ' 2026-03-20 16:26:25.717412 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.717416 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.717420 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.717424 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.717429 | controller | --timeout=300s' 2026-03-20 16:26:25.717433 | controller | - ' ' 2026-03-20 16:26:25.717437 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.717441 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.717448 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.717452 | controller | - ' ' 2026-03-20 16:26:25.717456 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.717460 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.717464 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.717469 | controller | - ' ' 2026-03-20 16:26:25.717473 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.717477 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.717481 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.717485 | controller | available"' 2026-03-20 16:26:25.717490 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.717503 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.717508 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.717514 | controller | - ' exit 1' 2026-03-20 16:26:25.717518 | controller | - ' fi' 2026-03-20 16:26:25.717523 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.717527 | controller | - ' ' 2026-03-20 16:26:25.717531 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.717535 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.717541 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.717547 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.717552 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.717558 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.717563 | controller | -- \' 2026-03-20 16:26:25.717567 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.717571 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.717575 | controller | - ' logger.go:42: 16:10:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.717580 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.717586 | controller | - ' logger.go:42: 16:10:14 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.717590 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.717594 | controller | found' 2026-03-20 16:26:25.717599 | controller | - ' logger.go:42: 16:10:15 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.717603 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.717607 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.717611 | controller | - ' ' 2026-03-20 16:26:25.717615 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.717620 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.717624 | controller | - ' ' 2026-03-20 16:26:25.717628 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.717632 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.717636 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.717641 | controller | - ' exit 1' 2026-03-20 16:26:25.717645 | controller | - ' fi' 2026-03-20 16:26:25.717649 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.717653 | controller | - ' ' 2026-03-20 16:26:25.717657 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.717662 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.717666 | controller | | base64 -d)' 2026-03-20 16:26:25.717670 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.717674 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.717678 | controller | - ' exit 1' 2026-03-20 16:26:25.717683 | controller | - ' fi' 2026-03-20 16:26:25.717687 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.717705 | controller | - ' ' 2026-03-20 16:26:25.717712 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.717716 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.717720 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.717724 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.717730 | controller | --timeout=300s' 2026-03-20 16:26:25.717735 | controller | - ' ' 2026-03-20 16:26:25.717739 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.717743 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.717747 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.717752 | controller | - ' ' 2026-03-20 16:26:25.717756 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.717760 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.717764 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.717768 | controller | - ' ' 2026-03-20 16:26:25.717772 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.717777 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.717781 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.717785 | controller | available"' 2026-03-20 16:26:25.717789 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.717794 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.717805 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.717810 | controller | - ' exit 1' 2026-03-20 16:26:25.717814 | controller | - ' fi' 2026-03-20 16:26:25.717819 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.717823 | controller | - ' ' 2026-03-20 16:26:25.717827 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.717831 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.717836 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.717840 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.717844 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.717848 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.717852 | controller | -- \' 2026-03-20 16:26:25.717857 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.717861 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.717865 | controller | - ' logger.go:42: 16:10:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.717869 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.717873 | controller | - ' logger.go:42: 16:10:15 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.717878 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.717882 | controller | found' 2026-03-20 16:26:25.717886 | controller | - ' logger.go:42: 16:10:16 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.717890 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.717895 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.717899 | controller | - ' ' 2026-03-20 16:26:25.717903 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.717907 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.717913 | controller | - ' ' 2026-03-20 16:26:25.717917 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.717922 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.717926 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.717930 | controller | - ' exit 1' 2026-03-20 16:26:25.717934 | controller | - ' fi' 2026-03-20 16:26:25.717938 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.717943 | controller | - ' ' 2026-03-20 16:26:25.717947 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.717951 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.717955 | controller | | base64 -d)' 2026-03-20 16:26:25.717960 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.717964 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.717968 | controller | - ' exit 1' 2026-03-20 16:26:25.717972 | controller | - ' fi' 2026-03-20 16:26:25.717976 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.717981 | controller | - ' ' 2026-03-20 16:26:25.717985 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.717989 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.717993 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.717997 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.718001 | controller | --timeout=300s' 2026-03-20 16:26:25.718008 | controller | - ' ' 2026-03-20 16:26:25.718012 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.718016 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.718020 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.718025 | controller | - ' ' 2026-03-20 16:26:25.718029 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.718033 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.718037 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.718041 | controller | - ' ' 2026-03-20 16:26:25.718045 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.718050 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.718054 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.718058 | controller | available"' 2026-03-20 16:26:25.718062 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.718067 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.718081 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.718087 | controller | - ' exit 1' 2026-03-20 16:26:25.718095 | controller | - ' fi' 2026-03-20 16:26:25.718100 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.718104 | controller | - ' ' 2026-03-20 16:26:25.718108 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.718112 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.718117 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.718123 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.718127 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.718131 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.718136 | controller | -- \' 2026-03-20 16:26:25.718140 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.718144 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.718148 | controller | - ' logger.go:42: 16:10:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.718154 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.718159 | controller | - ' logger.go:42: 16:10:17 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.718165 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.718171 | controller | found' 2026-03-20 16:26:25.718177 | controller | - ' logger.go:42: 16:10:18 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.718182 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.718190 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.718194 | controller | - ' ' 2026-03-20 16:26:25.718198 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.718203 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.718207 | controller | - ' ' 2026-03-20 16:26:25.718211 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.718215 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.718220 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.718226 | controller | - ' exit 1' 2026-03-20 16:26:25.718231 | controller | - ' fi' 2026-03-20 16:26:25.718237 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.718243 | controller | - ' ' 2026-03-20 16:26:25.718247 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.718251 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.718255 | controller | | base64 -d)' 2026-03-20 16:26:25.718259 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.718264 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.718268 | controller | - ' exit 1' 2026-03-20 16:26:25.718272 | controller | - ' fi' 2026-03-20 16:26:25.718276 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.718280 | controller | - ' ' 2026-03-20 16:26:25.718285 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.718289 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.718293 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.718297 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.718301 | controller | --timeout=300s' 2026-03-20 16:26:25.718305 | controller | - ' ' 2026-03-20 16:26:25.718309 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.718314 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.718319 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.718327 | controller | - ' ' 2026-03-20 16:26:25.718332 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.718338 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.718344 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.718349 | controller | - ' ' 2026-03-20 16:26:25.718354 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.718358 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.718362 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.718367 | controller | available"' 2026-03-20 16:26:25.718371 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.718375 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.718379 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.718383 | controller | - ' exit 1' 2026-03-20 16:26:25.718387 | controller | - ' fi' 2026-03-20 16:26:25.718401 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.718407 | controller | - ' ' 2026-03-20 16:26:25.718413 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.718418 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.718424 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.718430 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.718437 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.718441 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.718445 | controller | -- \' 2026-03-20 16:26:25.718450 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.718454 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.718458 | controller | - ' logger.go:42: 16:10:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.718462 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.718466 | controller | - ' logger.go:42: 16:10:18 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.718471 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.718475 | controller | found' 2026-03-20 16:26:25.718479 | controller | - ' logger.go:42: 16:10:19 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.718484 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.718490 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.718495 | controller | - ' ' 2026-03-20 16:26:25.718501 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.718507 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.718512 | controller | - ' ' 2026-03-20 16:26:25.718517 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.718522 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.718526 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.718530 | controller | - ' exit 1' 2026-03-20 16:26:25.718534 | controller | - ' fi' 2026-03-20 16:26:25.718539 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.718543 | controller | - ' ' 2026-03-20 16:26:25.718551 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.718555 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.718560 | controller | | base64 -d)' 2026-03-20 16:26:25.718564 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.718568 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.718572 | controller | - ' exit 1' 2026-03-20 16:26:25.718576 | controller | - ' fi' 2026-03-20 16:26:25.718581 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.718585 | controller | - ' ' 2026-03-20 16:26:25.718589 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.718593 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.718597 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.718602 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.718606 | controller | --timeout=300s' 2026-03-20 16:26:25.718612 | controller | - ' ' 2026-03-20 16:26:25.718623 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.718635 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.718641 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.718647 | controller | - ' ' 2026-03-20 16:26:25.718653 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.718659 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.718664 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.718670 | controller | - ' ' 2026-03-20 16:26:25.718675 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.718681 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.718686 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.718708 | controller | available"' 2026-03-20 16:26:25.718719 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.718725 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.718730 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.718736 | controller | - ' exit 1' 2026-03-20 16:26:25.718742 | controller | - ' fi' 2026-03-20 16:26:25.718760 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.718765 | controller | - ' ' 2026-03-20 16:26:25.718769 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.718774 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.718778 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.718782 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.718786 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.718791 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.718795 | controller | -- \' 2026-03-20 16:26:25.718799 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.718806 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.718811 | controller | - ' logger.go:42: 16:10:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.718815 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.718819 | controller | - ' logger.go:42: 16:10:19 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.718824 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.718828 | controller | found' 2026-03-20 16:26:25.718835 | controller | - ' logger.go:42: 16:10:20 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.718840 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.718844 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.718848 | controller | - ' ' 2026-03-20 16:26:25.718854 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.718859 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.718865 | controller | - ' ' 2026-03-20 16:26:25.718871 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.718877 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.718882 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.718887 | controller | - ' exit 1' 2026-03-20 16:26:25.718891 | controller | - ' fi' 2026-03-20 16:26:25.718895 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.718900 | controller | - ' ' 2026-03-20 16:26:25.718904 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.718908 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.718912 | controller | | base64 -d)' 2026-03-20 16:26:25.718917 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.718921 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.718925 | controller | - ' exit 1' 2026-03-20 16:26:25.718930 | controller | - ' fi' 2026-03-20 16:26:25.718934 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.718938 | controller | - ' ' 2026-03-20 16:26:25.718942 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.718946 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.718951 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.718955 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.718959 | controller | --timeout=300s' 2026-03-20 16:26:25.718963 | controller | - ' ' 2026-03-20 16:26:25.718967 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.718972 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.718976 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.718980 | controller | - ' ' 2026-03-20 16:26:25.718984 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.718989 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.718993 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.718997 | controller | - ' ' 2026-03-20 16:26:25.719001 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.719008 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.719012 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.719016 | controller | available"' 2026-03-20 16:26:25.719020 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.719025 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.719029 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.719033 | controller | - ' exit 1' 2026-03-20 16:26:25.719037 | controller | - ' fi' 2026-03-20 16:26:25.719041 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.719046 | controller | - ' ' 2026-03-20 16:26:25.719060 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.719065 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.719069 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.719073 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.719077 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.719082 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.719086 | controller | -- \' 2026-03-20 16:26:25.719090 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.719094 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.719098 | controller | - ' logger.go:42: 16:10:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.719103 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.719107 | controller | - ' logger.go:42: 16:10:20 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.719111 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.719115 | controller | found' 2026-03-20 16:26:25.719119 | controller | - ' logger.go:42: 16:10:21 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.719123 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.719128 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.719132 | controller | - ' ' 2026-03-20 16:26:25.719136 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.719140 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.719144 | controller | - ' ' 2026-03-20 16:26:25.719148 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.719153 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.719157 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.719163 | controller | - ' exit 1' 2026-03-20 16:26:25.719168 | controller | - ' fi' 2026-03-20 16:26:25.719172 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.719176 | controller | - ' ' 2026-03-20 16:26:25.719180 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.719185 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.719189 | controller | | base64 -d)' 2026-03-20 16:26:25.719193 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.719197 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.719201 | controller | - ' exit 1' 2026-03-20 16:26:25.719205 | controller | - ' fi' 2026-03-20 16:26:25.719212 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.719216 | controller | - ' ' 2026-03-20 16:26:25.719222 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.719226 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.719230 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.719235 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.719239 | controller | --timeout=300s' 2026-03-20 16:26:25.719243 | controller | - ' ' 2026-03-20 16:26:25.719248 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.719254 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.719267 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.719278 | controller | - ' ' 2026-03-20 16:26:25.719284 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.719291 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.719298 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.719306 | controller | - ' ' 2026-03-20 16:26:25.719312 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.719318 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.719324 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.719328 | controller | available"' 2026-03-20 16:26:25.719332 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.719337 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.719341 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.719345 | controller | - ' exit 1' 2026-03-20 16:26:25.719349 | controller | - ' fi' 2026-03-20 16:26:25.719354 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.719358 | controller | - ' ' 2026-03-20 16:26:25.719375 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.719380 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.719384 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.719389 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.719393 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.719397 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.719402 | controller | -- \' 2026-03-20 16:26:25.719406 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.719410 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.719414 | controller | - ' logger.go:42: 16:10:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.719419 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.719428 | controller | - ' logger.go:42: 16:10:21 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.719432 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.719439 | controller | found' 2026-03-20 16:26:25.719444 | controller | - ' logger.go:42: 16:10:22 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.719448 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.719453 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.719457 | controller | - ' ' 2026-03-20 16:26:25.719463 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.719467 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.719471 | controller | - ' ' 2026-03-20 16:26:25.719476 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.719480 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.719484 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.719488 | controller | - ' exit 1' 2026-03-20 16:26:25.719492 | controller | - ' fi' 2026-03-20 16:26:25.719497 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.719501 | controller | - ' ' 2026-03-20 16:26:25.719507 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.719511 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.719515 | controller | | base64 -d)' 2026-03-20 16:26:25.719520 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.719524 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.719528 | controller | - ' exit 1' 2026-03-20 16:26:25.719532 | controller | - ' fi' 2026-03-20 16:26:25.719536 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.719540 | controller | - ' ' 2026-03-20 16:26:25.719545 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.719549 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.719553 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.719557 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.719562 | controller | --timeout=300s' 2026-03-20 16:26:25.719568 | controller | - ' ' 2026-03-20 16:26:25.719574 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.719580 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.719586 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.719592 | controller | - ' ' 2026-03-20 16:26:25.719599 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.719607 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.719612 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.719617 | controller | - ' ' 2026-03-20 16:26:25.719638 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.719645 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.719650 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.719656 | controller | available"' 2026-03-20 16:26:25.719660 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.719664 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.719668 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.719672 | controller | - ' exit 1' 2026-03-20 16:26:25.719679 | controller | - ' fi' 2026-03-20 16:26:25.719683 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.719687 | controller | - ' ' 2026-03-20 16:26:25.719692 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.719723 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.719728 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.719732 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.719736 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.719740 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.719745 | controller | -- \' 2026-03-20 16:26:25.719749 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.719753 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.719757 | controller | - ' logger.go:42: 16:10:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.719762 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.719766 | controller | - ' logger.go:42: 16:10:22 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.719770 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.719774 | controller | found' 2026-03-20 16:26:25.719779 | controller | - ' logger.go:42: 16:10:23 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.719783 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.719787 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.719791 | controller | - ' ' 2026-03-20 16:26:25.719795 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.719800 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.719804 | controller | - ' ' 2026-03-20 16:26:25.719808 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.719812 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.719816 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.719821 | controller | - ' exit 1' 2026-03-20 16:26:25.719825 | controller | - ' fi' 2026-03-20 16:26:25.719829 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.719833 | controller | - ' ' 2026-03-20 16:26:25.719837 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.719842 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.719846 | controller | | base64 -d)' 2026-03-20 16:26:25.719850 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.719854 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.719858 | controller | - ' exit 1' 2026-03-20 16:26:25.719863 | controller | - ' fi' 2026-03-20 16:26:25.719867 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.719871 | controller | - ' ' 2026-03-20 16:26:25.719875 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.719880 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.719884 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.719888 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.719894 | controller | --timeout=300s' 2026-03-20 16:26:25.719898 | controller | - ' ' 2026-03-20 16:26:25.719903 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.719907 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.719911 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.719915 | controller | - ' ' 2026-03-20 16:26:25.719919 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.719924 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.719928 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.719932 | controller | - ' ' 2026-03-20 16:26:25.719936 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.719940 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.719945 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.719949 | controller | available"' 2026-03-20 16:26:25.719953 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.719957 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.719961 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.719965 | controller | - ' exit 1' 2026-03-20 16:26:25.719970 | controller | - ' fi' 2026-03-20 16:26:25.719974 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.719978 | controller | - ' ' 2026-03-20 16:26:25.719982 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.719993 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.719997 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.720001 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.720006 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.720010 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.720014 | controller | -- \' 2026-03-20 16:26:25.720018 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.720023 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.720027 | controller | - ' logger.go:42: 16:10:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.720034 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.720038 | controller | - ' logger.go:42: 16:10:24 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.720042 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.720046 | controller | found' 2026-03-20 16:26:25.720051 | controller | - ' logger.go:42: 16:10:25 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.720055 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.720059 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.720063 | controller | - ' ' 2026-03-20 16:26:25.720067 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.720075 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.720079 | controller | - ' ' 2026-03-20 16:26:25.720085 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.720089 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.720093 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.720097 | controller | - ' exit 1' 2026-03-20 16:26:25.720102 | controller | - ' fi' 2026-03-20 16:26:25.720106 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.720110 | controller | - ' ' 2026-03-20 16:26:25.720114 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.720118 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.720123 | controller | | base64 -d)' 2026-03-20 16:26:25.720127 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.720131 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.720135 | controller | - ' exit 1' 2026-03-20 16:26:25.720139 | controller | - ' fi' 2026-03-20 16:26:25.720144 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.720148 | controller | - ' ' 2026-03-20 16:26:25.720152 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.720156 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.720160 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.720165 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.720169 | controller | --timeout=300s' 2026-03-20 16:26:25.720173 | controller | - ' ' 2026-03-20 16:26:25.720177 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.720182 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.720186 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.720190 | controller | - ' ' 2026-03-20 16:26:25.720194 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.720198 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.720203 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.720207 | controller | - ' ' 2026-03-20 16:26:25.720211 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.720215 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.720219 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.720223 | controller | available"' 2026-03-20 16:26:25.720227 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.720232 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.720236 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.720240 | controller | - ' exit 1' 2026-03-20 16:26:25.720244 | controller | - ' fi' 2026-03-20 16:26:25.720248 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.720252 | controller | - ' ' 2026-03-20 16:26:25.720257 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.720261 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.720272 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.720279 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.720283 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.720287 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.720291 | controller | -- \' 2026-03-20 16:26:25.720295 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.720300 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.720304 | controller | - ' logger.go:42: 16:10:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.720308 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.720312 | controller | - ' logger.go:42: 16:10:25 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.720317 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.720321 | controller | found' 2026-03-20 16:26:25.720325 | controller | - ' logger.go:42: 16:10:26 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.720329 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.720333 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.720337 | controller | - ' ' 2026-03-20 16:26:25.720342 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.720346 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.720350 | controller | - ' ' 2026-03-20 16:26:25.720354 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.720358 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.720362 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.720367 | controller | - ' exit 1' 2026-03-20 16:26:25.720371 | controller | - ' fi' 2026-03-20 16:26:25.720378 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.720382 | controller | - ' ' 2026-03-20 16:26:25.720387 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.720391 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.720395 | controller | | base64 -d)' 2026-03-20 16:26:25.720399 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.720403 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.720407 | controller | - ' exit 1' 2026-03-20 16:26:25.720412 | controller | - ' fi' 2026-03-20 16:26:25.720416 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.720420 | controller | - ' ' 2026-03-20 16:26:25.720424 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.720428 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.720433 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.720437 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.720441 | controller | --timeout=300s' 2026-03-20 16:26:25.720445 | controller | - ' ' 2026-03-20 16:26:25.720449 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.720453 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.720458 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.720462 | controller | - ' ' 2026-03-20 16:26:25.720466 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.720635 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.720639 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.720643 | controller | - ' ' 2026-03-20 16:26:25.720648 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.720652 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.720658 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.720662 | controller | available"' 2026-03-20 16:26:25.720666 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.720670 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.720675 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.720679 | controller | - ' exit 1' 2026-03-20 16:26:25.720683 | controller | - ' fi' 2026-03-20 16:26:25.720687 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.720702 | controller | - ' ' 2026-03-20 16:26:25.720708 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.720713 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.720724 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.720729 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.720733 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.720737 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.720744 | controller | -- \' 2026-03-20 16:26:25.720748 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.720752 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.720756 | controller | - ' logger.go:42: 16:10:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.720760 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.720765 | controller | - ' logger.go:42: 16:10:26 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.720769 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.720773 | controller | found' 2026-03-20 16:26:25.720777 | controller | - ' logger.go:42: 16:10:27 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.720781 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.720785 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.720790 | controller | - ' ' 2026-03-20 16:26:25.720794 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.720798 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.720802 | controller | - ' ' 2026-03-20 16:26:25.720808 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.720812 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.720816 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.720820 | controller | - ' exit 1' 2026-03-20 16:26:25.720824 | controller | - ' fi' 2026-03-20 16:26:25.720829 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.720833 | controller | - ' ' 2026-03-20 16:26:25.720837 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.720843 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.720847 | controller | | base64 -d)' 2026-03-20 16:26:25.720851 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.720855 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.720860 | controller | - ' exit 1' 2026-03-20 16:26:25.720864 | controller | - ' fi' 2026-03-20 16:26:25.720868 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.720872 | controller | - ' ' 2026-03-20 16:26:25.720876 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.720881 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.720885 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.720889 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.720893 | controller | --timeout=300s' 2026-03-20 16:26:25.720897 | controller | - ' ' 2026-03-20 16:26:25.720902 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.720906 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.720910 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.720914 | controller | - ' ' 2026-03-20 16:26:25.720918 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.720922 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.720927 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.720931 | controller | - ' ' 2026-03-20 16:26:25.720935 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.720939 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.720943 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.720947 | controller | available"' 2026-03-20 16:26:25.720952 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.720958 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.720962 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.720966 | controller | - ' exit 1' 2026-03-20 16:26:25.720970 | controller | - ' fi' 2026-03-20 16:26:25.720975 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.720979 | controller | - ' ' 2026-03-20 16:26:25.720983 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.720987 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.720991 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.721006 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.721013 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.721018 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.721024 | controller | -- \' 2026-03-20 16:26:25.721030 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.721038 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.721043 | controller | - ' logger.go:42: 16:10:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.721049 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.721054 | controller | - ' logger.go:42: 16:10:27 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.721060 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.721065 | controller | found' 2026-03-20 16:26:25.721069 | controller | - ' logger.go:42: 16:10:28 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.721073 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.721077 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.721081 | controller | - ' ' 2026-03-20 16:26:25.721086 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.721090 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.721094 | controller | - ' ' 2026-03-20 16:26:25.721098 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.721102 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.721107 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.721111 | controller | - ' exit 1' 2026-03-20 16:26:25.721115 | controller | - ' fi' 2026-03-20 16:26:25.721119 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.721123 | controller | - ' ' 2026-03-20 16:26:25.721128 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.721132 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.721136 | controller | | base64 -d)' 2026-03-20 16:26:25.721140 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.721144 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.721148 | controller | - ' exit 1' 2026-03-20 16:26:25.721153 | controller | - ' fi' 2026-03-20 16:26:25.721158 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.721163 | controller | - ' ' 2026-03-20 16:26:25.721168 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.721174 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.721180 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.721186 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.721191 | controller | --timeout=300s' 2026-03-20 16:26:25.721197 | controller | - ' ' 2026-03-20 16:26:25.721202 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.721207 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.721213 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.721219 | controller | - ' ' 2026-03-20 16:26:25.721224 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.721230 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.721235 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.721239 | controller | - ' ' 2026-03-20 16:26:25.721244 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.721248 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.721255 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.721259 | controller | available"' 2026-03-20 16:26:25.721263 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.721267 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.721272 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.721276 | controller | - ' exit 1' 2026-03-20 16:26:25.721280 | controller | - ' fi' 2026-03-20 16:26:25.721284 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.721288 | controller | - ' ' 2026-03-20 16:26:25.721293 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.721297 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.721301 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.721314 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.721319 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.721323 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.721327 | controller | -- \' 2026-03-20 16:26:25.721331 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.721336 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.721340 | controller | - ' logger.go:42: 16:10:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.721344 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.721348 | controller | - ' logger.go:42: 16:10:28 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.721352 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.721356 | controller | found' 2026-03-20 16:26:25.721361 | controller | - ' logger.go:42: 16:10:29 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.721365 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.721369 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.721373 | controller | - ' ' 2026-03-20 16:26:25.721377 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.721382 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.721386 | controller | - ' ' 2026-03-20 16:26:25.721390 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.721394 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.721398 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.721403 | controller | - ' exit 1' 2026-03-20 16:26:25.721407 | controller | - ' fi' 2026-03-20 16:26:25.721411 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.721418 | controller | - ' ' 2026-03-20 16:26:25.721422 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.721426 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.721430 | controller | | base64 -d)' 2026-03-20 16:26:25.721435 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.721439 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.721443 | controller | - ' exit 1' 2026-03-20 16:26:25.721447 | controller | - ' fi' 2026-03-20 16:26:25.721451 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.721457 | controller | - ' ' 2026-03-20 16:26:25.721461 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.721466 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.721470 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.721474 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.721478 | controller | --timeout=300s' 2026-03-20 16:26:25.721483 | controller | - ' ' 2026-03-20 16:26:25.721487 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.721491 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.721495 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.721499 | controller | - ' ' 2026-03-20 16:26:25.721503 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.721508 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.721512 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.721516 | controller | - ' ' 2026-03-20 16:26:25.721520 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.721524 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.721529 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.721533 | controller | available"' 2026-03-20 16:26:25.721537 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.721541 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.721545 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.721549 | controller | - ' exit 1' 2026-03-20 16:26:25.721554 | controller | - ' fi' 2026-03-20 16:26:25.721560 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.721565 | controller | - ' ' 2026-03-20 16:26:25.721569 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.721573 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.721577 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.721581 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.721593 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.721597 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.721602 | controller | -- \' 2026-03-20 16:26:25.721606 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.721610 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.721614 | controller | - ' logger.go:42: 16:10:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.721618 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.721624 | controller | - ' logger.go:42: 16:10:29 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.721630 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.721635 | controller | found' 2026-03-20 16:26:25.721645 | controller | - ' logger.go:42: 16:10:30 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.721650 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.721655 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.721659 | controller | - ' ' 2026-03-20 16:26:25.721663 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.721667 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.721671 | controller | - ' ' 2026-03-20 16:26:25.721675 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.721680 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.721684 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.721688 | controller | - ' exit 1' 2026-03-20 16:26:25.721704 | controller | - ' fi' 2026-03-20 16:26:25.721711 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.721716 | controller | - ' ' 2026-03-20 16:26:25.721720 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.721724 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.721728 | controller | | base64 -d)' 2026-03-20 16:26:25.721732 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.721736 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.721741 | controller | - ' exit 1' 2026-03-20 16:26:25.721745 | controller | - ' fi' 2026-03-20 16:26:25.721749 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.721753 | controller | - ' ' 2026-03-20 16:26:25.721759 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.721764 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.721770 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.721775 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.721781 | controller | --timeout=300s' 2026-03-20 16:26:25.721785 | controller | - ' ' 2026-03-20 16:26:25.721789 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.721793 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.721798 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.721802 | controller | - ' ' 2026-03-20 16:26:25.721806 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.721810 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.721817 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.721821 | controller | - ' ' 2026-03-20 16:26:25.721825 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.721829 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.721833 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.721838 | controller | available"' 2026-03-20 16:26:25.721842 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.721846 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.721851 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.721856 | controller | - ' exit 1' 2026-03-20 16:26:25.721861 | controller | - ' fi' 2026-03-20 16:26:25.721870 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.721876 | controller | - ' ' 2026-03-20 16:26:25.721881 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.721887 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.721891 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.721895 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.721909 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.721913 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.721918 | controller | -- \' 2026-03-20 16:26:25.721922 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.721926 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.721930 | controller | - ' logger.go:42: 16:10:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.721934 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.721939 | controller | - ' logger.go:42: 16:10:30 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.721943 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.721947 | controller | found' 2026-03-20 16:26:25.721951 | controller | - ' logger.go:42: 16:10:31 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.721955 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.721959 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.721964 | controller | - ' ' 2026-03-20 16:26:25.721968 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.721972 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.721976 | controller | - ' ' 2026-03-20 16:26:25.721980 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.721984 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.721989 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.721993 | controller | - ' exit 1' 2026-03-20 16:26:25.721997 | controller | - ' fi' 2026-03-20 16:26:25.722001 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.722006 | controller | - ' ' 2026-03-20 16:26:25.722010 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.722014 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.722018 | controller | | base64 -d)' 2026-03-20 16:26:25.722022 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.722026 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.722031 | controller | - ' exit 1' 2026-03-20 16:26:25.722035 | controller | - ' fi' 2026-03-20 16:26:25.722039 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.722046 | controller | - ' ' 2026-03-20 16:26:25.722050 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.722054 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.722058 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.722062 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.722066 | controller | --timeout=300s' 2026-03-20 16:26:25.722071 | controller | - ' ' 2026-03-20 16:26:25.722078 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.722083 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.722087 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.722091 | controller | - ' ' 2026-03-20 16:26:25.722095 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.722099 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.722103 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.722108 | controller | - ' ' 2026-03-20 16:26:25.722112 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.722116 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.722120 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.722124 | controller | available"' 2026-03-20 16:26:25.722129 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.722133 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.722137 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.722141 | controller | - ' exit 1' 2026-03-20 16:26:25.722145 | controller | - ' fi' 2026-03-20 16:26:25.722149 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.722153 | controller | - ' ' 2026-03-20 16:26:25.722157 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.722162 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.722166 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.722170 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.722174 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.722186 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.722190 | controller | -- \' 2026-03-20 16:26:25.722194 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.722199 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.722203 | controller | - ' logger.go:42: 16:10:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.722207 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.722211 | controller | - ' logger.go:42: 16:10:32 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.722216 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.722220 | controller | found' 2026-03-20 16:26:25.722224 | controller | - ' logger.go:42: 16:10:33 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.722228 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.722232 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.722237 | controller | - ' ' 2026-03-20 16:26:25.722241 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.722245 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.722249 | controller | - ' ' 2026-03-20 16:26:25.722253 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.722260 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.722264 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.722268 | controller | - ' exit 1' 2026-03-20 16:26:25.722272 | controller | - ' fi' 2026-03-20 16:26:25.722276 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.722281 | controller | - ' ' 2026-03-20 16:26:25.722286 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.722291 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.722295 | controller | | base64 -d)' 2026-03-20 16:26:25.722301 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.722306 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.722310 | controller | - ' exit 1' 2026-03-20 16:26:25.722315 | controller | - ' fi' 2026-03-20 16:26:25.722320 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.722325 | controller | - ' ' 2026-03-20 16:26:25.722329 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.722333 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.722337 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.722342 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.722346 | controller | --timeout=300s' 2026-03-20 16:26:25.722352 | controller | - ' ' 2026-03-20 16:26:25.722356 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.722360 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.722364 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.722368 | controller | - ' ' 2026-03-20 16:26:25.722373 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.722377 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.722381 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.722385 | controller | - ' ' 2026-03-20 16:26:25.722389 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.722394 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.722398 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.722402 | controller | available"' 2026-03-20 16:26:25.722406 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.722410 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.722415 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.722419 | controller | - ' exit 1' 2026-03-20 16:26:25.722423 | controller | - ' fi' 2026-03-20 16:26:25.722427 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.722431 | controller | - ' ' 2026-03-20 16:26:25.722435 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.722440 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.722444 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.722448 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.722454 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.722465 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.722469 | controller | -- \' 2026-03-20 16:26:25.722474 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.722478 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.722482 | controller | - ' logger.go:42: 16:10:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.722486 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.722491 | controller | - ' logger.go:42: 16:10:33 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.722495 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.722499 | controller | found' 2026-03-20 16:26:25.722503 | controller | - ' logger.go:42: 16:10:34 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.722507 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.722511 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.722516 | controller | - ' ' 2026-03-20 16:26:25.722520 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.722524 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.722528 | controller | - ' ' 2026-03-20 16:26:25.722532 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.722536 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.722540 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.722545 | controller | - ' exit 1' 2026-03-20 16:26:25.722549 | controller | - ' fi' 2026-03-20 16:26:25.722555 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.722559 | controller | - ' ' 2026-03-20 16:26:25.722563 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.722568 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.722572 | controller | | base64 -d)' 2026-03-20 16:26:25.722576 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.722580 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.722584 | controller | - ' exit 1' 2026-03-20 16:26:25.722589 | controller | - ' fi' 2026-03-20 16:26:25.722593 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.722597 | controller | - ' ' 2026-03-20 16:26:25.722601 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.722605 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.722610 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.722615 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.722620 | controller | --timeout=300s' 2026-03-20 16:26:25.722624 | controller | - ' ' 2026-03-20 16:26:25.722629 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.722634 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.722638 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.722642 | controller | - ' ' 2026-03-20 16:26:25.722646 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.722652 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.722656 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.722661 | controller | - ' ' 2026-03-20 16:26:25.722665 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.722669 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.722673 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.722677 | controller | available"' 2026-03-20 16:26:25.722682 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.722686 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.722690 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.722708 | controller | - ' exit 1' 2026-03-20 16:26:25.722713 | controller | - ' fi' 2026-03-20 16:26:25.722717 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.722721 | controller | - ' ' 2026-03-20 16:26:25.722725 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.722729 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.722734 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.722738 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.722742 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.722746 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.722750 | controller | -- \' 2026-03-20 16:26:25.722762 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.722766 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.722773 | controller | - ' logger.go:42: 16:10:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.722777 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.722781 | controller | - ' logger.go:42: 16:10:34 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.722785 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.722790 | controller | found' 2026-03-20 16:26:25.722794 | controller | - ' logger.go:42: 16:10:35 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.722798 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.722802 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.722806 | controller | - ' ' 2026-03-20 16:26:25.722811 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.722815 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.722819 | controller | - ' ' 2026-03-20 16:26:25.722823 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.722827 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.722832 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.722836 | controller | - ' exit 1' 2026-03-20 16:26:25.722840 | controller | - ' fi' 2026-03-20 16:26:25.722844 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.722848 | controller | - ' ' 2026-03-20 16:26:25.722852 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.722857 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.722864 | controller | | base64 -d)' 2026-03-20 16:26:25.722869 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.722873 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.722877 | controller | - ' exit 1' 2026-03-20 16:26:25.722881 | controller | - ' fi' 2026-03-20 16:26:25.722885 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.722906 | controller | - ' ' 2026-03-20 16:26:25.722911 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.722915 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.722919 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.722923 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.722927 | controller | --timeout=300s' 2026-03-20 16:26:25.722931 | controller | - ' ' 2026-03-20 16:26:25.722936 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.722940 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.722944 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.722948 | controller | - ' ' 2026-03-20 16:26:25.722952 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.722956 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.722961 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.722965 | controller | - ' ' 2026-03-20 16:26:25.722969 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.722973 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.722977 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.722982 | controller | available"' 2026-03-20 16:26:25.722994 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.722999 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.723003 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.723007 | controller | - ' exit 1' 2026-03-20 16:26:25.723011 | controller | - ' fi' 2026-03-20 16:26:25.723016 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.723020 | controller | - ' ' 2026-03-20 16:26:25.723024 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.723028 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.723032 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.723036 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.723041 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.723045 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.723049 | controller | -- \' 2026-03-20 16:26:25.723061 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.723065 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.723080 | controller | - ' logger.go:42: 16:10:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.723084 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.723089 | controller | - ' logger.go:42: 16:10:35 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.723093 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.723097 | controller | found' 2026-03-20 16:26:25.723101 | controller | - ' logger.go:42: 16:10:36 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.723105 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.723110 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.723116 | controller | - ' ' 2026-03-20 16:26:25.723120 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.723125 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.723129 | controller | - ' ' 2026-03-20 16:26:25.723133 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.723137 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.723142 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.723146 | controller | - ' exit 1' 2026-03-20 16:26:25.723150 | controller | - ' fi' 2026-03-20 16:26:25.723154 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.723159 | controller | - ' ' 2026-03-20 16:26:25.723163 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.723174 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.723179 | controller | | base64 -d)' 2026-03-20 16:26:25.723183 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.723187 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.723191 | controller | - ' exit 1' 2026-03-20 16:26:25.723196 | controller | - ' fi' 2026-03-20 16:26:25.723200 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.723204 | controller | - ' ' 2026-03-20 16:26:25.723208 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.723212 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.723217 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.723221 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.723225 | controller | --timeout=300s' 2026-03-20 16:26:25.723229 | controller | - ' ' 2026-03-20 16:26:25.723233 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.723238 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.723242 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.723246 | controller | - ' ' 2026-03-20 16:26:25.723250 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.723255 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.723267 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.723271 | controller | - ' ' 2026-03-20 16:26:25.723277 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.723282 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.723286 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.723292 | controller | available"' 2026-03-20 16:26:25.723296 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.723300 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.723304 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.723309 | controller | - ' exit 1' 2026-03-20 16:26:25.723313 | controller | - ' fi' 2026-03-20 16:26:25.723317 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.723321 | controller | - ' ' 2026-03-20 16:26:25.723326 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.723330 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.723334 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.723338 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.723342 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.723347 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.723351 | controller | -- \' 2026-03-20 16:26:25.723355 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.723366 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.723371 | controller | - ' logger.go:42: 16:10:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.723375 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.723379 | controller | - ' logger.go:42: 16:10:36 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.723384 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.723388 | controller | found' 2026-03-20 16:26:25.723392 | controller | - ' logger.go:42: 16:10:37 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.723396 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.723400 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.723405 | controller | - ' ' 2026-03-20 16:26:25.723409 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.723413 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.723417 | controller | - ' ' 2026-03-20 16:26:25.723421 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.723425 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.723430 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.723436 | controller | - ' exit 1' 2026-03-20 16:26:25.723440 | controller | - ' fi' 2026-03-20 16:26:25.723444 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.723448 | controller | - ' ' 2026-03-20 16:26:25.723453 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.723457 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.723461 | controller | | base64 -d)' 2026-03-20 16:26:25.723465 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.723469 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.723473 | controller | - ' exit 1' 2026-03-20 16:26:25.723478 | controller | - ' fi' 2026-03-20 16:26:25.723482 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.723486 | controller | - ' ' 2026-03-20 16:26:25.723492 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.723496 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.723501 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.723505 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.723509 | controller | --timeout=300s' 2026-03-20 16:26:25.723513 | controller | - ' ' 2026-03-20 16:26:25.723517 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.723521 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.723525 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.723530 | controller | - ' ' 2026-03-20 16:26:25.723534 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.723538 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.723542 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.723546 | controller | - ' ' 2026-03-20 16:26:25.723550 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.723555 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.723559 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.723563 | controller | available"' 2026-03-20 16:26:25.723567 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.723573 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.723577 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.723581 | controller | - ' exit 1' 2026-03-20 16:26:25.723586 | controller | - ' fi' 2026-03-20 16:26:25.723590 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.723594 | controller | - ' ' 2026-03-20 16:26:25.723598 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.723602 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.723607 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.723611 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.723615 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.723619 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.723657 | controller | -- \' 2026-03-20 16:26:25.723662 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.723673 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.723678 | controller | - ' logger.go:42: 16:10:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.723682 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.723686 | controller | - ' logger.go:42: 16:10:37 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.723690 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.723707 | controller | found' 2026-03-20 16:26:25.723712 | controller | - ' logger.go:42: 16:10:38 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.723719 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.723723 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.723727 | controller | - ' ' 2026-03-20 16:26:25.723732 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.723736 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.723740 | controller | - ' ' 2026-03-20 16:26:25.723744 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.723748 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.723752 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.723756 | controller | - ' exit 1' 2026-03-20 16:26:25.723761 | controller | - ' fi' 2026-03-20 16:26:25.723765 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.723769 | controller | - ' ' 2026-03-20 16:26:25.723773 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.723777 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.723782 | controller | | base64 -d)' 2026-03-20 16:26:25.723788 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.723793 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.723797 | controller | - ' exit 1' 2026-03-20 16:26:25.723801 | controller | - ' fi' 2026-03-20 16:26:25.723805 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.723809 | controller | - ' ' 2026-03-20 16:26:25.723813 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.723818 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.723822 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.723826 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.723830 | controller | --timeout=300s' 2026-03-20 16:26:25.723834 | controller | - ' ' 2026-03-20 16:26:25.723838 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.723843 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.723847 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.723851 | controller | - ' ' 2026-03-20 16:26:25.723855 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.723859 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.723863 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.723868 | controller | - ' ' 2026-03-20 16:26:25.723872 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.723878 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.723882 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.723887 | controller | available"' 2026-03-20 16:26:25.723892 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.723896 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.723900 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.723904 | controller | - ' exit 1' 2026-03-20 16:26:25.723908 | controller | - ' fi' 2026-03-20 16:26:25.723913 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.723919 | controller | - ' ' 2026-03-20 16:26:25.723923 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.723928 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.723932 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.723936 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.723940 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.723944 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.723948 | controller | -- \' 2026-03-20 16:26:25.723953 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.723957 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.723969 | controller | - ' logger.go:42: 16:10:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.723973 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.723977 | controller | - ' logger.go:42: 16:10:38 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.723982 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.723986 | controller | found' 2026-03-20 16:26:25.723990 | controller | - ' logger.go:42: 16:10:39 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.723994 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.723999 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.724003 | controller | - ' ' 2026-03-20 16:26:25.724007 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.724011 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.724015 | controller | - ' ' 2026-03-20 16:26:25.724020 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.724024 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.724028 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.724032 | controller | - ' exit 1' 2026-03-20 16:26:25.724036 | controller | - ' fi' 2026-03-20 16:26:25.724040 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.724045 | controller | - ' ' 2026-03-20 16:26:25.724049 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.724053 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.724057 | controller | | base64 -d)' 2026-03-20 16:26:25.724061 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.724066 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.724070 | controller | - ' exit 1' 2026-03-20 16:26:25.724074 | controller | - ' fi' 2026-03-20 16:26:25.724078 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.724082 | controller | - ' ' 2026-03-20 16:26:25.724086 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.724091 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.724095 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.724099 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.724103 | controller | --timeout=300s' 2026-03-20 16:26:25.724107 | controller | - ' ' 2026-03-20 16:26:25.724111 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.724118 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.724131 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.724135 | controller | - ' ' 2026-03-20 16:26:25.724139 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.724143 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.724147 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.724152 | controller | - ' ' 2026-03-20 16:26:25.724156 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.724160 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.724164 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.724168 | controller | available"' 2026-03-20 16:26:25.724172 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.724177 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.724181 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.724185 | controller | - ' exit 1' 2026-03-20 16:26:25.724189 | controller | - ' fi' 2026-03-20 16:26:25.724193 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.724198 | controller | - ' ' 2026-03-20 16:26:25.724202 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.724206 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.724210 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.724214 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.724219 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.724223 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.724227 | controller | -- \' 2026-03-20 16:26:25.724231 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.724235 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.724246 | controller | - ' logger.go:42: 16:10:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.724251 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.724255 | controller | - ' logger.go:42: 16:10:40 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.724259 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.724263 | controller | found' 2026-03-20 16:26:25.724268 | controller | - ' logger.go:42: 16:10:41 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.724272 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.724276 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.724280 | controller | - ' ' 2026-03-20 16:26:25.724284 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.724288 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.724292 | controller | - ' ' 2026-03-20 16:26:25.724297 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.724303 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.724307 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.724311 | controller | - ' exit 1' 2026-03-20 16:26:25.724315 | controller | - ' fi' 2026-03-20 16:26:25.724320 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.724324 | controller | - ' ' 2026-03-20 16:26:25.724328 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.724332 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.724336 | controller | | base64 -d)' 2026-03-20 16:26:25.724340 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.724345 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.724349 | controller | - ' exit 1' 2026-03-20 16:26:25.724353 | controller | - ' fi' 2026-03-20 16:26:25.724357 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.724361 | controller | - ' ' 2026-03-20 16:26:25.724365 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.724370 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.724374 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.724378 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.724382 | controller | --timeout=300s' 2026-03-20 16:26:25.724386 | controller | - ' ' 2026-03-20 16:26:25.724390 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.724395 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.724399 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.724403 | controller | - ' ' 2026-03-20 16:26:25.724407 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.724411 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.724415 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.724419 | controller | - ' ' 2026-03-20 16:26:25.724424 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.724428 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.724432 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.724436 | controller | available"' 2026-03-20 16:26:25.724440 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.724448 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.724452 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.724456 | controller | - ' exit 1' 2026-03-20 16:26:25.724461 | controller | - ' fi' 2026-03-20 16:26:25.724465 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.724469 | controller | - ' ' 2026-03-20 16:26:25.724473 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.724477 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.724481 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.724486 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.724491 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.724495 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.724500 | controller | -- \' 2026-03-20 16:26:25.724504 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.724508 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.724512 | controller | - ' logger.go:42: 16:10:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.724524 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.724528 | controller | - ' logger.go:42: 16:10:41 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.724532 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.724537 | controller | found' 2026-03-20 16:26:25.724541 | controller | - ' logger.go:42: 16:10:42 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.724545 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.724549 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.724553 | controller | - ' ' 2026-03-20 16:26:25.724557 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.724562 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.724566 | controller | - ' ' 2026-03-20 16:26:25.724570 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.724574 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.724578 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.724583 | controller | - ' exit 1' 2026-03-20 16:26:25.724587 | controller | - ' fi' 2026-03-20 16:26:25.724591 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.724595 | controller | - ' ' 2026-03-20 16:26:25.724599 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.724604 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.724608 | controller | | base64 -d)' 2026-03-20 16:26:25.724612 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.724616 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.724620 | controller | - ' exit 1' 2026-03-20 16:26:25.724625 | controller | - ' fi' 2026-03-20 16:26:25.724629 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.724633 | controller | - ' ' 2026-03-20 16:26:25.724637 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.724641 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.724646 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.724650 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.724654 | controller | --timeout=300s' 2026-03-20 16:26:25.724658 | controller | - ' ' 2026-03-20 16:26:25.724662 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.724667 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.724671 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.724675 | controller | - ' ' 2026-03-20 16:26:25.724679 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.724683 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.724689 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.724705 | controller | - ' ' 2026-03-20 16:26:25.724710 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.724714 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.724720 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.724725 | controller | available"' 2026-03-20 16:26:25.724729 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.724733 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.724737 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.724741 | controller | - ' exit 1' 2026-03-20 16:26:25.724746 | controller | - ' fi' 2026-03-20 16:26:25.724750 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.724754 | controller | - ' ' 2026-03-20 16:26:25.724758 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.724762 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.724767 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.724771 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.724775 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.724779 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.724784 | controller | -- \' 2026-03-20 16:26:25.724788 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.724792 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.724796 | controller | - ' logger.go:42: 16:10:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.724800 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.724812 | controller | - ' logger.go:42: 16:10:42 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.724816 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.724821 | controller | found' 2026-03-20 16:26:25.724825 | controller | - ' logger.go:42: 16:10:43 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.724829 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.724833 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.724838 | controller | - ' ' 2026-03-20 16:26:25.724844 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.724848 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.724852 | controller | - ' ' 2026-03-20 16:26:25.724857 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.724861 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.724865 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.724869 | controller | - ' exit 1' 2026-03-20 16:26:25.724873 | controller | - ' fi' 2026-03-20 16:26:25.724878 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.724882 | controller | - ' ' 2026-03-20 16:26:25.724888 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.724892 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.724898 | controller | | base64 -d)' 2026-03-20 16:26:25.724902 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.724907 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.724911 | controller | - ' exit 1' 2026-03-20 16:26:25.724915 | controller | - ' fi' 2026-03-20 16:26:25.724919 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.724923 | controller | - ' ' 2026-03-20 16:26:25.724928 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.724932 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.724936 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.724940 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.724944 | controller | --timeout=300s' 2026-03-20 16:26:25.724950 | controller | - ' ' 2026-03-20 16:26:25.724954 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.724958 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.724964 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.724968 | controller | - ' ' 2026-03-20 16:26:25.724973 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.724977 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.724981 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.724985 | controller | - ' ' 2026-03-20 16:26:25.724989 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.724993 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.724997 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.725002 | controller | available"' 2026-03-20 16:26:25.725006 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.725010 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.725014 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.725018 | controller | - ' exit 1' 2026-03-20 16:26:25.725023 | controller | - ' fi' 2026-03-20 16:26:25.725027 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.725031 | controller | - ' ' 2026-03-20 16:26:25.725037 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.725041 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.725045 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.725049 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.725054 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.725058 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.725062 | controller | -- \' 2026-03-20 16:26:25.725066 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.725070 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.725074 | controller | - ' logger.go:42: 16:10:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.725081 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.725085 | controller | - ' logger.go:42: 16:10:43 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.725097 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.725101 | controller | found' 2026-03-20 16:26:25.725105 | controller | - ' logger.go:42: 16:10:44 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.725110 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.725114 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.725118 | controller | - ' ' 2026-03-20 16:26:25.725122 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.725126 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.725131 | controller | - ' ' 2026-03-20 16:26:25.725135 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.725139 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.725143 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.725147 | controller | - ' exit 1' 2026-03-20 16:26:25.725152 | controller | - ' fi' 2026-03-20 16:26:25.725156 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.725160 | controller | - ' ' 2026-03-20 16:26:25.725164 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.725168 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.725173 | controller | | base64 -d)' 2026-03-20 16:26:25.725177 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.725181 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.725185 | controller | - ' exit 1' 2026-03-20 16:26:25.725190 | controller | - ' fi' 2026-03-20 16:26:25.725194 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.725198 | controller | - ' ' 2026-03-20 16:26:25.725202 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.725206 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.725211 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.725215 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.725219 | controller | --timeout=300s' 2026-03-20 16:26:25.725223 | controller | - ' ' 2026-03-20 16:26:25.725227 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.725232 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.725236 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.725240 | controller | - ' ' 2026-03-20 16:26:25.725244 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.725249 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.725253 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.725257 | controller | - ' ' 2026-03-20 16:26:25.725261 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.725265 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.725270 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.725275 | controller | available"' 2026-03-20 16:26:25.725280 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.725284 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.725288 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.725292 | controller | - ' exit 1' 2026-03-20 16:26:25.725298 | controller | - ' fi' 2026-03-20 16:26:25.725303 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.725308 | controller | - ' ' 2026-03-20 16:26:25.725312 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.725317 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.725322 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.725327 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.725331 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.725335 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.725339 | controller | -- \' 2026-03-20 16:26:25.725344 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.725348 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.725352 | controller | - ' logger.go:42: 16:10:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.725356 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.725360 | controller | - ' logger.go:42: 16:10:44 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.725371 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.725376 | controller | found' 2026-03-20 16:26:25.725380 | controller | - ' logger.go:42: 16:10:45 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.725384 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.725389 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.725393 | controller | - ' ' 2026-03-20 16:26:25.725397 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.725401 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.725405 | controller | - ' ' 2026-03-20 16:26:25.725409 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.725413 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.725418 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.725422 | controller | - ' exit 1' 2026-03-20 16:26:25.725426 | controller | - ' fi' 2026-03-20 16:26:25.725430 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.725434 | controller | - ' ' 2026-03-20 16:26:25.725439 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.725443 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.725447 | controller | | base64 -d)' 2026-03-20 16:26:25.725451 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.725455 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.725459 | controller | - ' exit 1' 2026-03-20 16:26:25.725464 | controller | - ' fi' 2026-03-20 16:26:25.725468 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.725472 | controller | - ' ' 2026-03-20 16:26:25.725476 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.725482 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.725487 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.725491 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.725495 | controller | --timeout=300s' 2026-03-20 16:26:25.725499 | controller | - ' ' 2026-03-20 16:26:25.725503 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.725508 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.725512 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.725516 | controller | - ' ' 2026-03-20 16:26:25.725520 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.725524 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.725528 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.725533 | controller | - ' ' 2026-03-20 16:26:25.725537 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.725541 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.725545 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.725549 | controller | available"' 2026-03-20 16:26:25.725554 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.725558 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.725562 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.725566 | controller | - ' exit 1' 2026-03-20 16:26:25.725570 | controller | - ' fi' 2026-03-20 16:26:25.725575 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.725579 | controller | - ' ' 2026-03-20 16:26:25.725583 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.725587 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.725591 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.725597 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.725602 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.725606 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.725610 | controller | -- \' 2026-03-20 16:26:25.725614 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.725618 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.725622 | controller | - ' logger.go:42: 16:10:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.725627 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.725631 | controller | - ' logger.go:42: 16:10:45 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.725635 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.725639 | controller | found' 2026-03-20 16:26:25.725650 | controller | - ' logger.go:42: 16:10:46 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.725654 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.725661 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.725665 | controller | - ' ' 2026-03-20 16:26:25.725669 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.725673 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.725678 | controller | - ' ' 2026-03-20 16:26:25.725682 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.725686 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.725690 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.725707 | controller | - ' exit 1' 2026-03-20 16:26:25.725711 | controller | - ' fi' 2026-03-20 16:26:25.725715 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.725719 | controller | - ' ' 2026-03-20 16:26:25.725724 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.725728 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.725732 | controller | | base64 -d)' 2026-03-20 16:26:25.725736 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.725741 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.725745 | controller | - ' exit 1' 2026-03-20 16:26:25.725749 | controller | - ' fi' 2026-03-20 16:26:25.725753 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.725757 | controller | - ' ' 2026-03-20 16:26:25.725761 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.725766 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.725771 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.725777 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.725783 | controller | --timeout=300s' 2026-03-20 16:26:25.725788 | controller | - ' ' 2026-03-20 16:26:25.725794 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.725799 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.725805 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.725809 | controller | - ' ' 2026-03-20 16:26:25.725813 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.725817 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.725822 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.725826 | controller | - ' ' 2026-03-20 16:26:25.725834 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.725838 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.725843 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.725847 | controller | available"' 2026-03-20 16:26:25.725851 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.725855 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.725859 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.725863 | controller | - ' exit 1' 2026-03-20 16:26:25.725868 | controller | - ' fi' 2026-03-20 16:26:25.725872 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.725876 | controller | - ' ' 2026-03-20 16:26:25.725882 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.725887 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.725891 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.725895 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.725901 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.725905 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.725909 | controller | -- \' 2026-03-20 16:26:25.725913 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.725918 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.725922 | controller | - ' logger.go:42: 16:10:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.725926 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.725930 | controller | - ' logger.go:42: 16:10:46 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.725934 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.725938 | controller | found' 2026-03-20 16:26:25.725954 | controller | - ' logger.go:42: 16:10:47 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.725959 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.725963 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.725968 | controller | - ' ' 2026-03-20 16:26:25.725972 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.725976 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.725980 | controller | - ' ' 2026-03-20 16:26:25.725984 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.725988 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.725993 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.725997 | controller | - ' exit 1' 2026-03-20 16:26:25.726001 | controller | - ' fi' 2026-03-20 16:26:25.726005 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.726009 | controller | - ' ' 2026-03-20 16:26:25.726013 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.726018 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.726022 | controller | | base64 -d)' 2026-03-20 16:26:25.726026 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.726030 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.726034 | controller | - ' exit 1' 2026-03-20 16:26:25.726039 | controller | - ' fi' 2026-03-20 16:26:25.726043 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.726050 | controller | - ' ' 2026-03-20 16:26:25.726056 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.726062 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.726067 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.726073 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.726078 | controller | --timeout=300s' 2026-03-20 16:26:25.726084 | controller | - ' ' 2026-03-20 16:26:25.726090 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.726101 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.726107 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.726112 | controller | - ' ' 2026-03-20 16:26:25.726118 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.726123 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.726129 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.726134 | controller | - ' ' 2026-03-20 16:26:25.726139 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.726143 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.726147 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.726152 | controller | available"' 2026-03-20 16:26:25.726156 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.726162 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.726167 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.726171 | controller | - ' exit 1' 2026-03-20 16:26:25.726175 | controller | - ' fi' 2026-03-20 16:26:25.726179 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.726183 | controller | - ' ' 2026-03-20 16:26:25.726188 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.726192 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.726196 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.726200 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.726204 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.726209 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.726213 | controller | -- \' 2026-03-20 16:26:25.726217 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.726223 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.726227 | controller | - ' logger.go:42: 16:10:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.726231 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.726236 | controller | - ' logger.go:42: 16:10:48 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.726240 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.726244 | controller | found' 2026-03-20 16:26:25.726248 | controller | - ' logger.go:42: 16:10:49 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.726252 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.726265 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.726270 | controller | - ' ' 2026-03-20 16:26:25.726274 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.726278 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.726282 | controller | - ' ' 2026-03-20 16:26:25.726286 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.726291 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.726297 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.726301 | controller | - ' exit 1' 2026-03-20 16:26:25.726305 | controller | - ' fi' 2026-03-20 16:26:25.726309 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.726314 | controller | - ' ' 2026-03-20 16:26:25.726318 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.726322 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.726326 | controller | | base64 -d)' 2026-03-20 16:26:25.726330 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.726335 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.726339 | controller | - ' exit 1' 2026-03-20 16:26:25.726343 | controller | - ' fi' 2026-03-20 16:26:25.726347 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.726351 | controller | - ' ' 2026-03-20 16:26:25.726355 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.726360 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.726364 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.726368 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.726372 | controller | --timeout=300s' 2026-03-20 16:26:25.726379 | controller | - ' ' 2026-03-20 16:26:25.726383 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.726387 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.726535 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.726539 | controller | - ' ' 2026-03-20 16:26:25.726544 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.726548 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.726552 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.726556 | controller | - ' ' 2026-03-20 16:26:25.726560 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.726564 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.726569 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.726573 | controller | available"' 2026-03-20 16:26:25.726577 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.726581 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.726585 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.726589 | controller | - ' exit 1' 2026-03-20 16:26:25.726594 | controller | - ' fi' 2026-03-20 16:26:25.726598 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.726602 | controller | - ' ' 2026-03-20 16:26:25.726606 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.726610 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.726614 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.726619 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.726623 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.726629 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.726633 | controller | -- \' 2026-03-20 16:26:25.726637 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.726641 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.726646 | controller | - ' logger.go:42: 16:10:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.726650 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.726654 | controller | - ' logger.go:42: 16:10:49 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.726658 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.726662 | controller | found' 2026-03-20 16:26:25.726667 | controller | - ' logger.go:42: 16:10:50 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.726671 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.726675 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.726679 | controller | - ' ' 2026-03-20 16:26:25.726691 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.726709 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.726713 | controller | - ' ' 2026-03-20 16:26:25.726719 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.726724 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.726730 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.726736 | controller | - ' exit 1' 2026-03-20 16:26:25.726742 | controller | - ' fi' 2026-03-20 16:26:25.726748 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.726754 | controller | - ' ' 2026-03-20 16:26:25.726759 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.726765 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.726771 | controller | | base64 -d)' 2026-03-20 16:26:25.726775 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.726779 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.726783 | controller | - ' exit 1' 2026-03-20 16:26:25.726787 | controller | - ' fi' 2026-03-20 16:26:25.726792 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.726796 | controller | - ' ' 2026-03-20 16:26:25.726800 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.726804 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.726808 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.726812 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.726817 | controller | --timeout=300s' 2026-03-20 16:26:25.726821 | controller | - ' ' 2026-03-20 16:26:25.726825 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.726829 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.726833 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.726837 | controller | - ' ' 2026-03-20 16:26:25.726842 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.726846 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.726853 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.726857 | controller | - ' ' 2026-03-20 16:26:25.726861 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.726865 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.726869 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.726874 | controller | available"' 2026-03-20 16:26:25.726878 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.726882 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.726886 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.726890 | controller | - ' exit 1' 2026-03-20 16:26:25.726895 | controller | - ' fi' 2026-03-20 16:26:25.726899 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.726903 | controller | - ' ' 2026-03-20 16:26:25.726907 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.726911 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.726916 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.726920 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.726924 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.726928 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.726932 | controller | -- \' 2026-03-20 16:26:25.726936 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.726941 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.726945 | controller | - ' logger.go:42: 16:10:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.726949 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.726956 | controller | - ' logger.go:42: 16:10:50 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.726962 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.726966 | controller | found' 2026-03-20 16:26:25.726970 | controller | - ' logger.go:42: 16:10:51 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.726974 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.726979 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.726983 | controller | - ' ' 2026-03-20 16:26:25.726996 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.727001 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.727005 | controller | - ' ' 2026-03-20 16:26:25.727009 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.727013 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.727017 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.727022 | controller | - ' exit 1' 2026-03-20 16:26:25.727026 | controller | - ' fi' 2026-03-20 16:26:25.727030 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.727034 | controller | - ' ' 2026-03-20 16:26:25.727038 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.727043 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.727047 | controller | | base64 -d)' 2026-03-20 16:26:25.727053 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.727057 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.727061 | controller | - ' exit 1' 2026-03-20 16:26:25.727065 | controller | - ' fi' 2026-03-20 16:26:25.727071 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.727076 | controller | - ' ' 2026-03-20 16:26:25.727080 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.727084 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.727088 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.727092 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.727096 | controller | --timeout=300s' 2026-03-20 16:26:25.727101 | controller | - ' ' 2026-03-20 16:26:25.727105 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.727109 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.727113 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.727117 | controller | - ' ' 2026-03-20 16:26:25.727121 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.727129 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.727133 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.727137 | controller | - ' ' 2026-03-20 16:26:25.727141 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.727145 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.727150 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.727154 | controller | available"' 2026-03-20 16:26:25.727158 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.727162 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.727166 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.727170 | controller | - ' exit 1' 2026-03-20 16:26:25.727175 | controller | - ' fi' 2026-03-20 16:26:25.727179 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.727183 | controller | - ' ' 2026-03-20 16:26:25.727187 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.727191 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.727196 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.727200 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.727204 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.727208 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.727212 | controller | -- \' 2026-03-20 16:26:25.727217 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.727221 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.727225 | controller | - ' logger.go:42: 16:10:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.727231 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.727235 | controller | - ' logger.go:42: 16:10:51 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.727239 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.727243 | controller | found' 2026-03-20 16:26:25.727248 | controller | - ' logger.go:42: 16:10:52 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.727252 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.727256 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.727260 | controller | - ' ' 2026-03-20 16:26:25.727264 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.727275 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.727280 | controller | - ' ' 2026-03-20 16:26:25.727284 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.727288 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.727293 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.727297 | controller | - ' exit 1' 2026-03-20 16:26:25.727301 | controller | - ' fi' 2026-03-20 16:26:25.727305 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.727310 | controller | - ' ' 2026-03-20 16:26:25.727314 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.727318 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.727322 | controller | | base64 -d)' 2026-03-20 16:26:25.727326 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.727330 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.727335 | controller | - ' exit 1' 2026-03-20 16:26:25.727339 | controller | - ' fi' 2026-03-20 16:26:25.727343 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.727347 | controller | - ' ' 2026-03-20 16:26:25.727352 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.727356 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.727360 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.727364 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.727368 | controller | --timeout=300s' 2026-03-20 16:26:25.727372 | controller | - ' ' 2026-03-20 16:26:25.727377 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.727381 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.727385 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.727389 | controller | - ' ' 2026-03-20 16:26:25.727393 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.727397 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.727402 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.727406 | controller | - ' ' 2026-03-20 16:26:25.727410 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.727414 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.727418 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.727422 | controller | available"' 2026-03-20 16:26:25.727429 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.727433 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.727437 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.727441 | controller | - ' exit 1' 2026-03-20 16:26:25.727445 | controller | - ' fi' 2026-03-20 16:26:25.727449 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.727454 | controller | - ' ' 2026-03-20 16:26:25.727458 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.727462 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.727466 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.727470 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.727474 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.727479 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.727486 | controller | -- \' 2026-03-20 16:26:25.727490 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.727495 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.727499 | controller | - ' logger.go:42: 16:10:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.727503 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.727507 | controller | - ' logger.go:42: 16:10:52 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.727513 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.727517 | controller | found' 2026-03-20 16:26:25.727522 | controller | - ' logger.go:42: 16:10:53 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.727526 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.727530 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.727534 | controller | - ' ' 2026-03-20 16:26:25.727538 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.727558 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.727563 | controller | - ' ' 2026-03-20 16:26:25.727567 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.727572 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.727576 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.727582 | controller | - ' exit 1' 2026-03-20 16:26:25.727586 | controller | - ' fi' 2026-03-20 16:26:25.727591 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.727595 | controller | - ' ' 2026-03-20 16:26:25.727599 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.727605 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.727609 | controller | | base64 -d)' 2026-03-20 16:26:25.727613 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.727617 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.727622 | controller | - ' exit 1' 2026-03-20 16:26:25.727626 | controller | - ' fi' 2026-03-20 16:26:25.727682 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.727687 | controller | - ' ' 2026-03-20 16:26:25.727691 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.727708 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.727715 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.727719 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.727723 | controller | --timeout=300s' 2026-03-20 16:26:25.727727 | controller | - ' ' 2026-03-20 16:26:25.727732 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.727736 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.727740 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.727744 | controller | - ' ' 2026-03-20 16:26:25.727748 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.727753 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.727757 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.727761 | controller | - ' ' 2026-03-20 16:26:25.727765 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.727769 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.727773 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.727777 | controller | available"' 2026-03-20 16:26:25.727782 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.727786 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.727790 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.727794 | controller | - ' exit 1' 2026-03-20 16:26:25.727798 | controller | - ' fi' 2026-03-20 16:26:25.727803 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.727807 | controller | - ' ' 2026-03-20 16:26:25.727811 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.727815 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.727819 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.727823 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.727828 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.727832 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.727836 | controller | -- \' 2026-03-20 16:26:25.727840 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.727844 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.727848 | controller | - ' logger.go:42: 16:10:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.727853 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.727857 | controller | - ' logger.go:42: 16:10:53 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.727861 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.727865 | controller | found' 2026-03-20 16:26:25.727869 | controller | - ' logger.go:42: 16:10:54 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.727874 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.727878 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.727884 | controller | - ' ' 2026-03-20 16:26:25.727888 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.727894 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.727899 | controller | - ' ' 2026-03-20 16:26:25.727914 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.727921 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.727927 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.727933 | controller | - ' exit 1' 2026-03-20 16:26:25.727938 | controller | - ' fi' 2026-03-20 16:26:25.727944 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.727950 | controller | - ' ' 2026-03-20 16:26:25.727955 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.727960 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.727964 | controller | | base64 -d)' 2026-03-20 16:26:25.727969 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.727973 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.727977 | controller | - ' exit 1' 2026-03-20 16:26:25.727981 | controller | - ' fi' 2026-03-20 16:26:25.727985 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.727990 | controller | - ' ' 2026-03-20 16:26:25.727994 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.727998 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.728002 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.728006 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.728011 | controller | --timeout=300s' 2026-03-20 16:26:25.728015 | controller | - ' ' 2026-03-20 16:26:25.728019 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.728023 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.728027 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.728031 | controller | - ' ' 2026-03-20 16:26:25.728036 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.728040 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.728044 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.728048 | controller | - ' ' 2026-03-20 16:26:25.728052 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.728056 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.728061 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.728065 | controller | available"' 2026-03-20 16:26:25.728069 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.728073 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.728077 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.728082 | controller | - ' exit 1' 2026-03-20 16:26:25.728086 | controller | - ' fi' 2026-03-20 16:26:25.728090 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.728094 | controller | - ' ' 2026-03-20 16:26:25.728098 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.728105 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.728109 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.728113 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.728117 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.728122 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.728129 | controller | -- \' 2026-03-20 16:26:25.728134 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.728140 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.728145 | controller | - ' logger.go:42: 16:10:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.728151 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.728156 | controller | - ' logger.go:42: 16:10:54 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.728162 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.728166 | controller | found' 2026-03-20 16:26:25.728183 | controller | - ' logger.go:42: 16:10:55 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.728187 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.728191 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.728196 | controller | - ' ' 2026-03-20 16:26:25.728200 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.728204 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.728208 | controller | - ' ' 2026-03-20 16:26:25.728221 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.728226 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.728230 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.728234 | controller | - ' exit 1' 2026-03-20 16:26:25.728241 | controller | - ' fi' 2026-03-20 16:26:25.728245 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.728249 | controller | - ' ' 2026-03-20 16:26:25.728253 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.728258 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.728262 | controller | | base64 -d)' 2026-03-20 16:26:25.728266 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.728270 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.728274 | controller | - ' exit 1' 2026-03-20 16:26:25.728279 | controller | - ' fi' 2026-03-20 16:26:25.728283 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.728287 | controller | - ' ' 2026-03-20 16:26:25.728291 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.728295 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.728299 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.728304 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.728308 | controller | --timeout=300s' 2026-03-20 16:26:25.728312 | controller | - ' ' 2026-03-20 16:26:25.728316 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.728321 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.728327 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.728331 | controller | - ' ' 2026-03-20 16:26:25.728336 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.728340 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.728344 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.728348 | controller | - ' ' 2026-03-20 16:26:25.728352 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.728357 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.728361 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.728365 | controller | available"' 2026-03-20 16:26:25.728369 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.728373 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.728378 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.728382 | controller | - ' exit 1' 2026-03-20 16:26:25.728386 | controller | - ' fi' 2026-03-20 16:26:25.728390 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.728394 | controller | - ' ' 2026-03-20 16:26:25.728399 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.728403 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.728407 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.728411 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.728415 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.728420 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.728424 | controller | -- \' 2026-03-20 16:26:25.728428 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.728432 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.728436 | controller | - ' logger.go:42: 16:10:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.728441 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.728445 | controller | - ' logger.go:42: 16:10:56 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.728449 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.728453 | controller | found' 2026-03-20 16:26:25.728457 | controller | - ' logger.go:42: 16:10:57 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.728462 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.728466 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.728470 | controller | - ' ' 2026-03-20 16:26:25.728474 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.728479 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.728483 | controller | - ' ' 2026-03-20 16:26:25.728487 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.728498 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.728502 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.728508 | controller | - ' exit 1' 2026-03-20 16:26:25.728513 | controller | - ' fi' 2026-03-20 16:26:25.728517 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.728521 | controller | - ' ' 2026-03-20 16:26:25.728525 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.728529 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.728534 | controller | | base64 -d)' 2026-03-20 16:26:25.728538 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.728542 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.728546 | controller | - ' exit 1' 2026-03-20 16:26:25.728550 | controller | - ' fi' 2026-03-20 16:26:25.728555 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.728559 | controller | - ' ' 2026-03-20 16:26:25.728563 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.728567 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.728571 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.728575 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.728580 | controller | --timeout=300s' 2026-03-20 16:26:25.728584 | controller | - ' ' 2026-03-20 16:26:25.728588 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.728592 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.728597 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.728601 | controller | - ' ' 2026-03-20 16:26:25.728605 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.728609 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.728613 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.728618 | controller | - ' ' 2026-03-20 16:26:25.728622 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.728626 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.728630 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.728634 | controller | available"' 2026-03-20 16:26:25.728639 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.728643 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.728647 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.728651 | controller | - ' exit 1' 2026-03-20 16:26:25.728655 | controller | - ' fi' 2026-03-20 16:26:25.728660 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.728664 | controller | - ' ' 2026-03-20 16:26:25.728668 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.728672 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.728676 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.728680 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.728685 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.728704 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.728711 | controller | -- \' 2026-03-20 16:26:25.728715 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.728719 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.728723 | controller | - ' logger.go:42: 16:10:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.728728 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.728732 | controller | - ' logger.go:42: 16:10:57 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.728736 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.728740 | controller | found' 2026-03-20 16:26:25.728744 | controller | - ' logger.go:42: 16:10:58 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.728748 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.728753 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.728757 | controller | - ' ' 2026-03-20 16:26:25.728763 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.728767 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.728772 | controller | - ' ' 2026-03-20 16:26:25.728776 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.728780 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.728792 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.728796 | controller | - ' exit 1' 2026-03-20 16:26:25.728800 | controller | - ' fi' 2026-03-20 16:26:25.728805 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.728809 | controller | - ' ' 2026-03-20 16:26:25.728813 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.728817 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.728821 | controller | | base64 -d)' 2026-03-20 16:26:25.728825 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.728830 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.728834 | controller | - ' exit 1' 2026-03-20 16:26:25.728838 | controller | - ' fi' 2026-03-20 16:26:25.728842 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.728846 | controller | - ' ' 2026-03-20 16:26:25.728850 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.728855 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.728859 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.728863 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.728867 | controller | --timeout=300s' 2026-03-20 16:26:25.728871 | controller | - ' ' 2026-03-20 16:26:25.728876 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.728880 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.728884 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.728888 | controller | - ' ' 2026-03-20 16:26:25.728892 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.728896 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.728901 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.728906 | controller | - ' ' 2026-03-20 16:26:25.728911 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.728915 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.728921 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.728926 | controller | available"' 2026-03-20 16:26:25.728930 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.728934 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.728940 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.728944 | controller | - ' exit 1' 2026-03-20 16:26:25.728948 | controller | - ' fi' 2026-03-20 16:26:25.728952 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.728957 | controller | - ' ' 2026-03-20 16:26:25.728961 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.728965 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.728969 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.728973 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.728978 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.728982 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.728986 | controller | -- \' 2026-03-20 16:26:25.728990 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.728994 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.728999 | controller | - ' logger.go:42: 16:10:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.729003 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.729007 | controller | - ' logger.go:42: 16:10:58 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.729011 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.729015 | controller | found' 2026-03-20 16:26:25.729020 | controller | - ' logger.go:42: 16:10:59 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.729024 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.729028 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.729032 | controller | - ' ' 2026-03-20 16:26:25.729036 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.729040 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.729045 | controller | - ' ' 2026-03-20 16:26:25.729049 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.729053 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.729057 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.729061 | controller | - ' exit 1' 2026-03-20 16:26:25.729074 | controller | - ' fi' 2026-03-20 16:26:25.729078 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.729082 | controller | - ' ' 2026-03-20 16:26:25.729087 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.729091 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.729095 | controller | | base64 -d)' 2026-03-20 16:26:25.729099 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.729105 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.729109 | controller | - ' exit 1' 2026-03-20 16:26:25.729114 | controller | - ' fi' 2026-03-20 16:26:25.729118 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.729122 | controller | - ' ' 2026-03-20 16:26:25.729126 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.729130 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.729135 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.729139 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.729143 | controller | --timeout=300s' 2026-03-20 16:26:25.729147 | controller | - ' ' 2026-03-20 16:26:25.729151 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.729155 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.729160 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.729164 | controller | - ' ' 2026-03-20 16:26:25.729168 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.729172 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.729178 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.729182 | controller | - ' ' 2026-03-20 16:26:25.729187 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.729191 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.729195 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.729199 | controller | available"' 2026-03-20 16:26:25.729203 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.729208 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.729212 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.729216 | controller | - ' exit 1' 2026-03-20 16:26:25.729222 | controller | - ' fi' 2026-03-20 16:26:25.729226 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.729230 | controller | - ' ' 2026-03-20 16:26:25.729234 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.729238 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.729243 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.729247 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.729251 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.729255 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.729259 | controller | -- \' 2026-03-20 16:26:25.729263 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.729268 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.729272 | controller | - ' logger.go:42: 16:10:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.729276 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.729282 | controller | - ' logger.go:42: 16:10:59 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.729286 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.729290 | controller | found' 2026-03-20 16:26:25.729294 | controller | - ' logger.go:42: 16:11:00 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.729298 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.729303 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.729309 | controller | - ' ' 2026-03-20 16:26:25.729314 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.729320 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.729325 | controller | - ' ' 2026-03-20 16:26:25.729330 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.729335 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.729339 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.729343 | controller | - ' exit 1' 2026-03-20 16:26:25.729347 | controller | - ' fi' 2026-03-20 16:26:25.729359 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.729363 | controller | - ' ' 2026-03-20 16:26:25.729367 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.729372 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.729376 | controller | | base64 -d)' 2026-03-20 16:26:25.729380 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.729384 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.729388 | controller | - ' exit 1' 2026-03-20 16:26:25.729393 | controller | - ' fi' 2026-03-20 16:26:25.729397 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.729401 | controller | - ' ' 2026-03-20 16:26:25.729405 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.729409 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.729414 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.729418 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.729422 | controller | --timeout=300s' 2026-03-20 16:26:25.729426 | controller | - ' ' 2026-03-20 16:26:25.729430 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.729434 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.729439 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.729443 | controller | - ' ' 2026-03-20 16:26:25.729447 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.729451 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.729455 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.729459 | controller | - ' ' 2026-03-20 16:26:25.729464 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.729468 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.729472 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.729476 | controller | available"' 2026-03-20 16:26:25.729480 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.729486 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.729490 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.729495 | controller | - ' exit 1' 2026-03-20 16:26:25.729499 | controller | - ' fi' 2026-03-20 16:26:25.729503 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.729507 | controller | - ' ' 2026-03-20 16:26:25.729511 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.729515 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.729519 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.729524 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.729528 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.729532 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.729536 | controller | -- \' 2026-03-20 16:26:25.729540 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.729544 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.729549 | controller | - ' logger.go:42: 16:11:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.729553 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.729557 | controller | - ' logger.go:42: 16:11:00 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.729561 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.729565 | controller | found' 2026-03-20 16:26:25.729569 | controller | - ' logger.go:42: 16:11:01 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.729574 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.729578 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.729582 | controller | - ' ' 2026-03-20 16:26:25.729586 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.729590 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.729594 | controller | - ' ' 2026-03-20 16:26:25.729599 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.729603 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.729607 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.729611 | controller | - ' exit 1' 2026-03-20 16:26:25.729615 | controller | - ' fi' 2026-03-20 16:26:25.729619 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.729624 | controller | - ' ' 2026-03-20 16:26:25.729634 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.729638 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.729643 | controller | | base64 -d)' 2026-03-20 16:26:25.729647 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.729651 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.729655 | controller | - ' exit 1' 2026-03-20 16:26:25.729660 | controller | - ' fi' 2026-03-20 16:26:25.729664 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.729668 | controller | - ' ' 2026-03-20 16:26:25.729672 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.729676 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.729682 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.729686 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.729691 | controller | --timeout=300s' 2026-03-20 16:26:25.729707 | controller | - ' ' 2026-03-20 16:26:25.729712 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.729716 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.729720 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.729724 | controller | - ' ' 2026-03-20 16:26:25.729728 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.729732 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.729737 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.729741 | controller | - ' ' 2026-03-20 16:26:25.729745 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.729749 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.729753 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.729757 | controller | available"' 2026-03-20 16:26:25.729762 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.729766 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.729770 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.729774 | controller | - ' exit 1' 2026-03-20 16:26:25.729778 | controller | - ' fi' 2026-03-20 16:26:25.729782 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.729787 | controller | - ' ' 2026-03-20 16:26:25.729791 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.729795 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.729799 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.729803 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.729807 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.729812 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.729816 | controller | -- \' 2026-03-20 16:26:25.729820 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.729824 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.729829 | controller | - ' logger.go:42: 16:11:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.729833 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.729837 | controller | - ' logger.go:42: 16:11:01 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.729843 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.729848 | controller | found' 2026-03-20 16:26:25.729852 | controller | - ' logger.go:42: 16:11:02 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.729856 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.729860 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.729864 | controller | - ' ' 2026-03-20 16:26:25.729869 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.729876 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.729880 | controller | - ' ' 2026-03-20 16:26:25.729886 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.729890 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.729895 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.729899 | controller | - ' exit 1' 2026-03-20 16:26:25.729903 | controller | - ' fi' 2026-03-20 16:26:25.729907 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.729911 | controller | - ' ' 2026-03-20 16:26:25.729923 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.729927 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.729931 | controller | | base64 -d)' 2026-03-20 16:26:25.729936 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.729940 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.729944 | controller | - ' exit 1' 2026-03-20 16:26:25.729948 | controller | - ' fi' 2026-03-20 16:26:25.729952 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.729957 | controller | - ' ' 2026-03-20 16:26:25.729961 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.729965 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.729969 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.729973 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.729978 | controller | --timeout=300s' 2026-03-20 16:26:25.729982 | controller | - ' ' 2026-03-20 16:26:25.729986 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.729990 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.729994 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.729998 | controller | - ' ' 2026-03-20 16:26:25.730003 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.730007 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.730011 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.730015 | controller | - ' ' 2026-03-20 16:26:25.730019 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.730023 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.730028 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.730032 | controller | available"' 2026-03-20 16:26:25.730036 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.730040 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.730044 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.730048 | controller | - ' exit 1' 2026-03-20 16:26:25.730053 | controller | - ' fi' 2026-03-20 16:26:25.730057 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.730061 | controller | - ' ' 2026-03-20 16:26:25.730065 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.730073 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.730077 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.730081 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.730085 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.730089 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.730094 | controller | -- \' 2026-03-20 16:26:25.730098 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.730102 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.730108 | controller | - ' logger.go:42: 16:11:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.730112 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.730117 | controller | - ' logger.go:42: 16:11:03 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.730121 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.730125 | controller | found' 2026-03-20 16:26:25.730129 | controller | - ' logger.go:42: 16:11:04 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.730133 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.730138 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.730142 | controller | - ' ' 2026-03-20 16:26:25.730146 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.730150 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.730154 | controller | - ' ' 2026-03-20 16:26:25.730159 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.730163 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.730167 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.730171 | controller | - ' exit 1' 2026-03-20 16:26:25.730175 | controller | - ' fi' 2026-03-20 16:26:25.730179 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.730184 | controller | - ' ' 2026-03-20 16:26:25.730188 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.730199 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.730204 | controller | | base64 -d)' 2026-03-20 16:26:25.730208 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.730212 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.730218 | controller | - ' exit 1' 2026-03-20 16:26:25.730223 | controller | - ' fi' 2026-03-20 16:26:25.730227 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.730231 | controller | - ' ' 2026-03-20 16:26:25.730235 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.730239 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.730244 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.730248 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.730252 | controller | --timeout=300s' 2026-03-20 16:26:25.730256 | controller | - ' ' 2026-03-20 16:26:25.730262 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.730268 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.730274 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.730278 | controller | - ' ' 2026-03-20 16:26:25.730282 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.730286 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.730290 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.730295 | controller | - ' ' 2026-03-20 16:26:25.730299 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.730303 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.730307 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.730311 | controller | available"' 2026-03-20 16:26:25.730316 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.730320 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.730324 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.730328 | controller | - ' exit 1' 2026-03-20 16:26:25.730332 | controller | - ' fi' 2026-03-20 16:26:25.730337 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.730341 | controller | - ' ' 2026-03-20 16:26:25.730345 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.730349 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.730353 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.730357 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.730362 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.730366 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.730370 | controller | -- \' 2026-03-20 16:26:25.730374 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.730379 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.730383 | controller | - ' logger.go:42: 16:11:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.730387 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.730391 | controller | - ' logger.go:42: 16:11:04 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.730395 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.730399 | controller | found' 2026-03-20 16:26:25.730404 | controller | - ' logger.go:42: 16:11:05 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.730408 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.730412 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.730418 | controller | - ' ' 2026-03-20 16:26:25.730423 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.730427 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.730431 | controller | - ' ' 2026-03-20 16:26:25.730435 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.730439 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.730444 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.730448 | controller | - ' exit 1' 2026-03-20 16:26:25.730454 | controller | - ' fi' 2026-03-20 16:26:25.730458 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.730462 | controller | - ' ' 2026-03-20 16:26:25.730467 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.730471 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.730482 | controller | | base64 -d)' 2026-03-20 16:26:25.730487 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.730491 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.730495 | controller | - ' exit 1' 2026-03-20 16:26:25.730500 | controller | - ' fi' 2026-03-20 16:26:25.730504 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.730508 | controller | - ' ' 2026-03-20 16:26:25.730512 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.730519 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.730523 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.730527 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.730531 | controller | --timeout=300s' 2026-03-20 16:26:25.730536 | controller | - ' ' 2026-03-20 16:26:25.730540 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.730544 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.730550 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.730555 | controller | - ' ' 2026-03-20 16:26:25.730561 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.730567 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.730572 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.730578 | controller | - ' ' 2026-03-20 16:26:25.730583 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.730587 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.730592 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.730596 | controller | available"' 2026-03-20 16:26:25.730600 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.730604 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.730608 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.730613 | controller | - ' exit 1' 2026-03-20 16:26:25.730617 | controller | - ' fi' 2026-03-20 16:26:25.730621 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.730626 | controller | - ' ' 2026-03-20 16:26:25.730630 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.730634 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.730638 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.730642 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.730647 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.730651 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.730657 | controller | -- \' 2026-03-20 16:26:25.730661 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.730666 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.730670 | controller | - ' logger.go:42: 16:11:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.730674 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.730732 | controller | - ' logger.go:42: 16:11:05 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.730738 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.730744 | controller | found' 2026-03-20 16:26:25.730750 | controller | - ' logger.go:42: 16:11:06 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.730756 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.730761 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.730767 | controller | - ' ' 2026-03-20 16:26:25.730773 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.730786 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.730790 | controller | - ' ' 2026-03-20 16:26:25.730794 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.730799 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.730803 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.730807 | controller | - ' exit 1' 2026-03-20 16:26:25.730812 | controller | - ' fi' 2026-03-20 16:26:25.730816 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.730820 | controller | - ' ' 2026-03-20 16:26:25.730824 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.730828 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.730833 | controller | | base64 -d)' 2026-03-20 16:26:25.730845 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.730850 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.730854 | controller | - ' exit 1' 2026-03-20 16:26:25.730858 | controller | - ' fi' 2026-03-20 16:26:25.730900 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.730905 | controller | - ' ' 2026-03-20 16:26:25.730919 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.730924 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.730928 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.730932 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.730936 | controller | --timeout=300s' 2026-03-20 16:26:25.730941 | controller | - ' ' 2026-03-20 16:26:25.730945 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.730949 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.730982 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.730988 | controller | - ' ' 2026-03-20 16:26:25.730994 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.731000 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.731006 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.731015 | controller | - ' ' 2026-03-20 16:26:25.731020 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.731024 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.731028 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.731032 | controller | available"' 2026-03-20 16:26:25.731036 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.731041 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.731054 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.731059 | controller | - ' exit 1' 2026-03-20 16:26:25.731063 | controller | - ' fi' 2026-03-20 16:26:25.731068 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.731072 | controller | - ' ' 2026-03-20 16:26:25.731076 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.731080 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.731085 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.731089 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.731093 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.731097 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.731102 | controller | -- \' 2026-03-20 16:26:25.731106 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.731110 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.731114 | controller | - ' logger.go:42: 16:11:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.731119 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.731123 | controller | - ' logger.go:42: 16:11:06 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.731127 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.731131 | controller | found' 2026-03-20 16:26:25.731136 | controller | - ' logger.go:42: 16:11:07 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.731140 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.731144 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.731148 | controller | - ' ' 2026-03-20 16:26:25.731153 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.731157 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.731161 | controller | - ' ' 2026-03-20 16:26:25.731165 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.731169 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.731174 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.731178 | controller | - ' exit 1' 2026-03-20 16:26:25.731182 | controller | - ' fi' 2026-03-20 16:26:25.731186 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.731191 | controller | - ' ' 2026-03-20 16:26:25.731195 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.731199 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.731203 | controller | | base64 -d)' 2026-03-20 16:26:25.731207 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.731220 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.731228 | controller | - ' exit 1' 2026-03-20 16:26:25.731232 | controller | - ' fi' 2026-03-20 16:26:25.731236 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.731241 | controller | - ' ' 2026-03-20 16:26:25.731245 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.731249 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.731256 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.731260 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.731264 | controller | --timeout=300s' 2026-03-20 16:26:25.731269 | controller | - ' ' 2026-03-20 16:26:25.731273 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.731277 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.731281 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.731285 | controller | - ' ' 2026-03-20 16:26:25.731290 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.731294 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.731298 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.731302 | controller | - ' ' 2026-03-20 16:26:25.731307 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.731312 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.731317 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.731321 | controller | available"' 2026-03-20 16:26:25.731325 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.731329 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.731333 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.731338 | controller | - ' exit 1' 2026-03-20 16:26:25.731342 | controller | - ' fi' 2026-03-20 16:26:25.731346 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.731350 | controller | - ' ' 2026-03-20 16:26:25.731354 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.731358 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.731363 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.731367 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.731371 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.731375 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.731379 | controller | -- \' 2026-03-20 16:26:25.731384 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.731388 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.731392 | controller | - ' logger.go:42: 16:11:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.731396 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.731400 | controller | - ' logger.go:42: 16:11:07 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.731408 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.731412 | controller | found' 2026-03-20 16:26:25.731416 | controller | - ' logger.go:42: 16:11:08 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.731421 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.731425 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.731429 | controller | - ' ' 2026-03-20 16:26:25.731433 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.731437 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.731442 | controller | - ' ' 2026-03-20 16:26:25.731446 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.731450 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.731454 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.731459 | controller | - ' exit 1' 2026-03-20 16:26:25.731463 | controller | - ' fi' 2026-03-20 16:26:25.731467 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.731471 | controller | - ' ' 2026-03-20 16:26:25.731478 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.731483 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.731487 | controller | | base64 -d)' 2026-03-20 16:26:25.731491 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.731503 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.731508 | controller | - ' exit 1' 2026-03-20 16:26:25.731512 | controller | - ' fi' 2026-03-20 16:26:25.731517 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.731521 | controller | - ' ' 2026-03-20 16:26:25.731525 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.731529 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.731533 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.731538 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.731542 | controller | --timeout=300s' 2026-03-20 16:26:25.731546 | controller | - ' ' 2026-03-20 16:26:25.731550 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.731554 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.731559 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.731563 | controller | - ' ' 2026-03-20 16:26:25.731567 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.731571 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.731575 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.731580 | controller | - ' ' 2026-03-20 16:26:25.731584 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.731588 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.731592 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.731597 | controller | available"' 2026-03-20 16:26:25.731601 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.731607 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.731613 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.731617 | controller | - ' exit 1' 2026-03-20 16:26:25.731621 | controller | - ' fi' 2026-03-20 16:26:25.731625 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.731630 | controller | - ' ' 2026-03-20 16:26:25.731660 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.731664 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.731669 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.731673 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.731677 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.731681 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.731685 | controller | -- \' 2026-03-20 16:26:25.731690 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.731709 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.731726 | controller | - ' logger.go:42: 16:11:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.731732 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.731737 | controller | - ' logger.go:42: 16:11:08 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.731741 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.731745 | controller | found' 2026-03-20 16:26:25.731750 | controller | - ' logger.go:42: 16:11:09 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.731754 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.731758 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.731762 | controller | - ' ' 2026-03-20 16:26:25.731767 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.731771 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.731775 | controller | - ' ' 2026-03-20 16:26:25.731779 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.731783 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.731788 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.731795 | controller | - ' exit 1' 2026-03-20 16:26:25.731799 | controller | - ' fi' 2026-03-20 16:26:25.731803 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.731808 | controller | - ' ' 2026-03-20 16:26:25.731812 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.731816 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.731820 | controller | | base64 -d)' 2026-03-20 16:26:25.731825 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.731829 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.731833 | controller | - ' exit 1' 2026-03-20 16:26:25.731845 | controller | - ' fi' 2026-03-20 16:26:25.731850 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.731854 | controller | - ' ' 2026-03-20 16:26:25.731858 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.731863 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.731867 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.731875 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.731879 | controller | --timeout=300s' 2026-03-20 16:26:25.731884 | controller | - ' ' 2026-03-20 16:26:25.731888 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.731892 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.731896 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.731900 | controller | - ' ' 2026-03-20 16:26:25.731905 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.731909 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.731913 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.731917 | controller | - ' ' 2026-03-20 16:26:25.731922 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.731926 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.731932 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.731937 | controller | available"' 2026-03-20 16:26:25.731943 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.731948 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.731954 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.731959 | controller | - ' exit 1' 2026-03-20 16:26:25.731963 | controller | - ' fi' 2026-03-20 16:26:25.731968 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.731972 | controller | - ' ' 2026-03-20 16:26:25.731976 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.731980 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.731984 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.731988 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.731992 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.731997 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.732001 | controller | -- \' 2026-03-20 16:26:25.732005 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.732009 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.732013 | controller | - ' logger.go:42: 16:11:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.732018 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.732022 | controller | - ' logger.go:42: 16:11:10 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.732026 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.732030 | controller | found' 2026-03-20 16:26:25.732043 | controller | - ' logger.go:42: 16:11:11 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.732047 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.732052 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.732056 | controller | - ' ' 2026-03-20 16:26:25.732060 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.732066 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.732070 | controller | - ' ' 2026-03-20 16:26:25.732075 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.732079 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.732083 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.732087 | controller | - ' exit 1' 2026-03-20 16:26:25.732092 | controller | - ' fi' 2026-03-20 16:26:25.732096 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.732100 | controller | - ' ' 2026-03-20 16:26:25.732104 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.732108 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.732113 | controller | | base64 -d)' 2026-03-20 16:26:25.732117 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.732121 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.732125 | controller | - ' exit 1' 2026-03-20 16:26:25.732130 | controller | - ' fi' 2026-03-20 16:26:25.732141 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.732146 | controller | - ' ' 2026-03-20 16:26:25.732150 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.732154 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.732159 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.732163 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.732167 | controller | --timeout=300s' 2026-03-20 16:26:25.732171 | controller | - ' ' 2026-03-20 16:26:25.732175 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.732179 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.732184 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.732188 | controller | - ' ' 2026-03-20 16:26:25.732192 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.732196 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.732200 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.732204 | controller | - ' ' 2026-03-20 16:26:25.732209 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.732213 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.732217 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.732221 | controller | available"' 2026-03-20 16:26:25.732225 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.732229 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.732234 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.732238 | controller | - ' exit 1' 2026-03-20 16:26:25.732242 | controller | - ' fi' 2026-03-20 16:26:25.732246 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.732250 | controller | - ' ' 2026-03-20 16:26:25.732254 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.732259 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.732265 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.732269 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.732273 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.732277 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.732281 | controller | -- \' 2026-03-20 16:26:25.732285 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.732290 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.732294 | controller | - ' logger.go:42: 16:11:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.732298 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.732303 | controller | - ' logger.go:42: 16:11:11 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.732307 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.732311 | controller | found' 2026-03-20 16:26:25.732315 | controller | - ' logger.go:42: 16:11:12 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.732319 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.732323 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.732328 | controller | - ' ' 2026-03-20 16:26:25.732332 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.732336 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.732340 | controller | - ' ' 2026-03-20 16:26:25.732344 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.732348 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.732353 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.732357 | controller | - ' exit 1' 2026-03-20 16:26:25.732361 | controller | - ' fi' 2026-03-20 16:26:25.732365 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.732370 | controller | - ' ' 2026-03-20 16:26:25.732376 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.732381 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.732385 | controller | | base64 -d)' 2026-03-20 16:26:25.732390 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.732395 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.732401 | controller | - ' exit 1' 2026-03-20 16:26:25.732406 | controller | - ' fi' 2026-03-20 16:26:25.732412 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.732417 | controller | - ' ' 2026-03-20 16:26:25.732429 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.732434 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.732438 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.732442 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.732447 | controller | --timeout=300s' 2026-03-20 16:26:25.732451 | controller | - ' ' 2026-03-20 16:26:25.732455 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.732459 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.732466 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.732472 | controller | - ' ' 2026-03-20 16:26:25.732476 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.732481 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.732485 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.732489 | controller | - ' ' 2026-03-20 16:26:25.732495 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.732499 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.732503 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.732507 | controller | available"' 2026-03-20 16:26:25.732511 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.732516 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.732520 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.732524 | controller | - ' exit 1' 2026-03-20 16:26:25.732528 | controller | - ' fi' 2026-03-20 16:26:25.732532 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.732537 | controller | - ' ' 2026-03-20 16:26:25.732543 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.732547 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.732551 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.732555 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.732560 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.732564 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.732568 | controller | -- \' 2026-03-20 16:26:25.732572 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.732576 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.732581 | controller | - ' logger.go:42: 16:11:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.732585 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.732589 | controller | - ' logger.go:42: 16:11:12 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.732593 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.732598 | controller | found' 2026-03-20 16:26:25.732602 | controller | - ' logger.go:42: 16:11:13 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.732606 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.732610 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.732614 | controller | - ' ' 2026-03-20 16:26:25.732619 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.732623 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.732627 | controller | - ' ' 2026-03-20 16:26:25.732631 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.732635 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.732640 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.732644 | controller | - ' exit 1' 2026-03-20 16:26:25.732648 | controller | - ' fi' 2026-03-20 16:26:25.732652 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.732658 | controller | - ' ' 2026-03-20 16:26:25.732662 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.732667 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.732671 | controller | | base64 -d)' 2026-03-20 16:26:25.732675 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.732679 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.732683 | controller | - ' exit 1' 2026-03-20 16:26:25.732687 | controller | - ' fi' 2026-03-20 16:26:25.732703 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.732710 | controller | - ' ' 2026-03-20 16:26:25.732714 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.732726 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.732731 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.732735 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.732739 | controller | --timeout=300s' 2026-03-20 16:26:25.732743 | controller | - ' ' 2026-03-20 16:26:25.732748 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.732752 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.732756 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.732760 | controller | - ' ' 2026-03-20 16:26:25.732764 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.732769 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.732773 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.732777 | controller | - ' ' 2026-03-20 16:26:25.732781 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.732785 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.732790 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.732794 | controller | available"' 2026-03-20 16:26:25.732798 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.732802 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.732806 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.732811 | controller | - ' exit 1' 2026-03-20 16:26:25.732815 | controller | - ' fi' 2026-03-20 16:26:25.732819 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.732823 | controller | - ' ' 2026-03-20 16:26:25.732827 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.732832 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.732836 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.732840 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.732844 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.732848 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.732852 | controller | -- \' 2026-03-20 16:26:25.732860 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.732864 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.732868 | controller | - ' logger.go:42: 16:11:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.732872 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.732876 | controller | - ' logger.go:42: 16:11:13 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.732881 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.732885 | controller | found' 2026-03-20 16:26:25.732889 | controller | - ' logger.go:42: 16:11:14 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.732893 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.732897 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.732902 | controller | - ' ' 2026-03-20 16:26:25.732906 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.732910 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.732914 | controller | - ' ' 2026-03-20 16:26:25.732918 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.732923 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.732927 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.732931 | controller | - ' exit 1' 2026-03-20 16:26:25.732935 | controller | - ' fi' 2026-03-20 16:26:25.732939 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.732944 | controller | - ' ' 2026-03-20 16:26:25.732950 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.732954 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.732959 | controller | | base64 -d)' 2026-03-20 16:26:25.732963 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.732967 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.732971 | controller | - ' exit 1' 2026-03-20 16:26:25.732975 | controller | - ' fi' 2026-03-20 16:26:25.732980 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.732984 | controller | - ' ' 2026-03-20 16:26:25.732988 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.732999 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.733003 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.733007 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.733012 | controller | --timeout=300s' 2026-03-20 16:26:25.733016 | controller | - ' ' 2026-03-20 16:26:25.733020 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.733024 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.733029 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.733033 | controller | - ' ' 2026-03-20 16:26:25.733037 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.733041 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.733045 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.733049 | controller | - ' ' 2026-03-20 16:26:25.733055 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.733060 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.733066 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.733070 | controller | available"' 2026-03-20 16:26:25.733074 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.733079 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.733084 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.733089 | controller | - ' exit 1' 2026-03-20 16:26:25.733093 | controller | - ' fi' 2026-03-20 16:26:25.733097 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.733101 | controller | - ' ' 2026-03-20 16:26:25.733106 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.733110 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.733114 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.733118 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.733122 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.733126 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.733131 | controller | -- \' 2026-03-20 16:26:25.733135 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.733139 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.733143 | controller | - ' logger.go:42: 16:11:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.733147 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.733152 | controller | - ' logger.go:42: 16:11:14 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.733156 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.733160 | controller | found' 2026-03-20 16:26:25.733164 | controller | - ' logger.go:42: 16:11:15 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.733168 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.733172 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.733177 | controller | - ' ' 2026-03-20 16:26:25.733181 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.733185 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.733189 | controller | - ' ' 2026-03-20 16:26:25.733193 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.733198 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.733202 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.733206 | controller | - ' exit 1' 2026-03-20 16:26:25.733210 | controller | - ' fi' 2026-03-20 16:26:25.733214 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.733219 | controller | - ' ' 2026-03-20 16:26:25.733223 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.733227 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.733231 | controller | | base64 -d)' 2026-03-20 16:26:25.733235 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.733239 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.733245 | controller | - ' exit 1' 2026-03-20 16:26:25.733249 | controller | - ' fi' 2026-03-20 16:26:25.733254 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.733258 | controller | - ' ' 2026-03-20 16:26:25.733262 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.733266 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.733277 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.733282 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.733286 | controller | --timeout=300s' 2026-03-20 16:26:25.733290 | controller | - ' ' 2026-03-20 16:26:25.733294 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.733299 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.733303 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.733307 | controller | - ' ' 2026-03-20 16:26:25.733311 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.733315 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.733319 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.733324 | controller | - ' ' 2026-03-20 16:26:25.733328 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.733332 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.733336 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.733340 | controller | available"' 2026-03-20 16:26:25.733345 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.733349 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.733353 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.733357 | controller | - ' exit 1' 2026-03-20 16:26:25.733361 | controller | - ' fi' 2026-03-20 16:26:25.733365 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.733370 | controller | - ' ' 2026-03-20 16:26:25.733374 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.733378 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.733382 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.733386 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.733392 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.733397 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.733401 | controller | -- \' 2026-03-20 16:26:25.733405 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.733409 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.733413 | controller | - ' logger.go:42: 16:11:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.733418 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.733422 | controller | - ' logger.go:42: 16:11:15 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.733428 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.733432 | controller | found' 2026-03-20 16:26:25.733436 | controller | - ' logger.go:42: 16:11:16 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.733441 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.733445 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.733449 | controller | - ' ' 2026-03-20 16:26:25.733453 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.733457 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.733462 | controller | - ' ' 2026-03-20 16:26:25.733466 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.733470 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.733474 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.733478 | controller | - ' exit 1' 2026-03-20 16:26:25.733483 | controller | - ' fi' 2026-03-20 16:26:25.733487 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.733491 | controller | - ' ' 2026-03-20 16:26:25.733495 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.733499 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.733503 | controller | | base64 -d)' 2026-03-20 16:26:25.733508 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.733512 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.733516 | controller | - ' exit 1' 2026-03-20 16:26:25.733520 | controller | - ' fi' 2026-03-20 16:26:25.733525 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.733529 | controller | - ' ' 2026-03-20 16:26:25.733533 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.733537 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.733547 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.733552 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.733556 | controller | --timeout=300s' 2026-03-20 16:26:25.733560 | controller | - ' ' 2026-03-20 16:26:25.733565 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.733569 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.733573 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.733577 | controller | - ' ' 2026-03-20 16:26:25.733584 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.733588 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.733592 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.733596 | controller | - ' ' 2026-03-20 16:26:25.733601 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.733605 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.733609 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.733613 | controller | available"' 2026-03-20 16:26:25.733617 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.733623 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.733627 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.733633 | controller | - ' exit 1' 2026-03-20 16:26:25.733637 | controller | - ' fi' 2026-03-20 16:26:25.733642 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.733646 | controller | - ' ' 2026-03-20 16:26:25.733650 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.733654 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.733658 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.733662 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.733667 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.733671 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.733675 | controller | -- \' 2026-03-20 16:26:25.733679 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.733683 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.733688 | controller | - ' logger.go:42: 16:11:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.733702 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.733709 | controller | - ' logger.go:42: 16:11:16 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.733713 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.733717 | controller | found' 2026-03-20 16:26:25.733721 | controller | - ' logger.go:42: 16:11:17 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.733725 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.733730 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.733734 | controller | - ' ' 2026-03-20 16:26:25.733738 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.733742 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.733746 | controller | - ' ' 2026-03-20 16:26:25.733751 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.733755 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.733759 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.733763 | controller | - ' exit 1' 2026-03-20 16:26:25.733768 | controller | - ' fi' 2026-03-20 16:26:25.733772 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.733776 | controller | - ' ' 2026-03-20 16:26:25.733780 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.733784 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.733788 | controller | | base64 -d)' 2026-03-20 16:26:25.733793 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.733797 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.733801 | controller | - ' exit 1' 2026-03-20 16:26:25.733805 | controller | - ' fi' 2026-03-20 16:26:25.733809 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.733814 | controller | - ' ' 2026-03-20 16:26:25.733818 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.733822 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.733826 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.733837 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.733844 | controller | --timeout=300s' 2026-03-20 16:26:25.733848 | controller | - ' ' 2026-03-20 16:26:25.733852 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.733856 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.733861 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.733865 | controller | - ' ' 2026-03-20 16:26:25.733869 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.733873 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.733878 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.733882 | controller | - ' ' 2026-03-20 16:26:25.733886 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.733890 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.733894 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.733899 | controller | available"' 2026-03-20 16:26:25.733903 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.733907 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.733911 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.733915 | controller | - ' exit 1' 2026-03-20 16:26:25.733920 | controller | - ' fi' 2026-03-20 16:26:25.733924 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.733928 | controller | - ' ' 2026-03-20 16:26:25.733932 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.733936 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.733941 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.733945 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.733949 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.733953 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.733958 | controller | -- \' 2026-03-20 16:26:25.733962 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.733966 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.733970 | controller | - ' logger.go:42: 16:11:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.733974 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.733979 | controller | - ' logger.go:42: 16:11:18 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.733983 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.733987 | controller | found' 2026-03-20 16:26:25.733991 | controller | - ' logger.go:42: 16:11:19 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.733995 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.733999 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.734004 | controller | - ' ' 2026-03-20 16:26:25.734008 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.734012 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.734018 | controller | - ' ' 2026-03-20 16:26:25.734022 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.734026 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.734030 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.734035 | controller | - ' exit 1' 2026-03-20 16:26:25.734039 | controller | - ' fi' 2026-03-20 16:26:25.734043 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.734047 | controller | - ' ' 2026-03-20 16:26:25.734051 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.734056 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.734060 | controller | | base64 -d)' 2026-03-20 16:26:25.734064 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.734068 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.734072 | controller | - ' exit 1' 2026-03-20 16:26:25.734077 | controller | - ' fi' 2026-03-20 16:26:25.734081 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.734085 | controller | - ' ' 2026-03-20 16:26:25.734089 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.734094 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.734098 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.734102 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.734113 | controller | --timeout=300s' 2026-03-20 16:26:25.734117 | controller | - ' ' 2026-03-20 16:26:25.734122 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.734126 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.734130 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.734134 | controller | - ' ' 2026-03-20 16:26:25.734139 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.734143 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.734147 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.734151 | controller | - ' ' 2026-03-20 16:26:25.734155 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.734160 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.734164 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.734168 | controller | available"' 2026-03-20 16:26:25.734172 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.734179 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.734183 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.734187 | controller | - ' exit 1' 2026-03-20 16:26:25.734191 | controller | - ' fi' 2026-03-20 16:26:25.734195 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.734200 | controller | - ' ' 2026-03-20 16:26:25.734205 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.734209 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.734214 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.734221 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.734225 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.734229 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.734234 | controller | -- \' 2026-03-20 16:26:25.734238 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.734242 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.734246 | controller | - ' logger.go:42: 16:11:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.734250 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.734257 | controller | - ' logger.go:42: 16:11:19 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.734261 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.734265 | controller | found' 2026-03-20 16:26:25.734269 | controller | - ' logger.go:42: 16:11:20 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.734274 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.734278 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.734282 | controller | - ' ' 2026-03-20 16:26:25.734287 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.734291 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.734295 | controller | - ' ' 2026-03-20 16:26:25.734299 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.734303 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.734307 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.734312 | controller | - ' exit 1' 2026-03-20 16:26:25.734316 | controller | - ' fi' 2026-03-20 16:26:25.734320 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.734324 | controller | - ' ' 2026-03-20 16:26:25.734328 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.734333 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.734337 | controller | | base64 -d)' 2026-03-20 16:26:25.734341 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.734345 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.734349 | controller | - ' exit 1' 2026-03-20 16:26:25.734353 | controller | - ' fi' 2026-03-20 16:26:25.734358 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.734364 | controller | - ' ' 2026-03-20 16:26:25.734368 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.734372 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.734376 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.734380 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.734385 | controller | --timeout=300s' 2026-03-20 16:26:25.734389 | controller | - ' ' 2026-03-20 16:26:25.734400 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.734405 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.734409 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.734415 | controller | - ' ' 2026-03-20 16:26:25.734419 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.734425 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.734429 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.734433 | controller | - ' ' 2026-03-20 16:26:25.734438 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.734442 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.734446 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.734450 | controller | available"' 2026-03-20 16:26:25.734454 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.734459 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.734463 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.734467 | controller | - ' exit 1' 2026-03-20 16:26:25.734471 | controller | - ' fi' 2026-03-20 16:26:25.734475 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.734480 | controller | - ' ' 2026-03-20 16:26:25.734484 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.734488 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.734492 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.734496 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.734500 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.734504 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.734509 | controller | -- \' 2026-03-20 16:26:25.734513 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.734517 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.734521 | controller | - ' logger.go:42: 16:11:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.734525 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.734530 | controller | - ' logger.go:42: 16:11:20 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.734534 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.734538 | controller | found' 2026-03-20 16:26:25.734542 | controller | - ' logger.go:42: 16:11:21 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.734546 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.734550 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.734555 | controller | - ' ' 2026-03-20 16:26:25.734559 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.734563 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.734567 | controller | - ' ' 2026-03-20 16:26:25.734571 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.734576 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.734580 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.734584 | controller | - ' exit 1' 2026-03-20 16:26:25.734588 | controller | - ' fi' 2026-03-20 16:26:25.734592 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.734598 | controller | - ' ' 2026-03-20 16:26:25.734602 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.734607 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.734611 | controller | | base64 -d)' 2026-03-20 16:26:25.734615 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.734619 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.734624 | controller | - ' exit 1' 2026-03-20 16:26:25.734817 | controller | - ' fi' 2026-03-20 16:26:25.734822 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.734826 | controller | - ' ' 2026-03-20 16:26:25.734830 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.734834 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.734838 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.734842 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.734847 | controller | --timeout=300s' 2026-03-20 16:26:25.734853 | controller | - ' ' 2026-03-20 16:26:25.734864 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.734870 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.734874 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.734879 | controller | - ' ' 2026-03-20 16:26:25.734883 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.734887 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.734891 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.734896 | controller | - ' ' 2026-03-20 16:26:25.734900 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.734904 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.734908 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.734912 | controller | available"' 2026-03-20 16:26:25.734916 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.734921 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.734925 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.734929 | controller | - ' exit 1' 2026-03-20 16:26:25.734933 | controller | - ' fi' 2026-03-20 16:26:25.734937 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.734942 | controller | - ' ' 2026-03-20 16:26:25.734946 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.734950 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.734954 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.734958 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.734962 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.734967 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.734971 | controller | -- \' 2026-03-20 16:26:25.734975 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.734981 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.734985 | controller | - ' logger.go:42: 16:11:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.734989 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.734995 | controller | - ' logger.go:42: 16:11:21 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.734999 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.735003 | controller | found' 2026-03-20 16:26:25.735008 | controller | - ' logger.go:42: 16:11:22 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.735012 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.735016 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.735020 | controller | - ' ' 2026-03-20 16:26:25.735024 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.735029 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.735033 | controller | - ' ' 2026-03-20 16:26:25.735037 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.735041 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.735045 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.735050 | controller | - ' exit 1' 2026-03-20 16:26:25.735054 | controller | - ' fi' 2026-03-20 16:26:25.735058 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.735062 | controller | - ' ' 2026-03-20 16:26:25.735066 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.735071 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.735075 | controller | | base64 -d)' 2026-03-20 16:26:25.735079 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.735083 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.735087 | controller | - ' exit 1' 2026-03-20 16:26:25.735092 | controller | - ' fi' 2026-03-20 16:26:25.735096 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.735100 | controller | - ' ' 2026-03-20 16:26:25.735104 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.735108 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.735113 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.735117 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.735121 | controller | --timeout=300s' 2026-03-20 16:26:25.735125 | controller | - ' ' 2026-03-20 16:26:25.735129 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.735140 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.735145 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.735149 | controller | - ' ' 2026-03-20 16:26:25.735153 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.735157 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.735161 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.735166 | controller | - ' ' 2026-03-20 16:26:25.735170 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.735176 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.735180 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.735184 | controller | available"' 2026-03-20 16:26:25.735188 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.735192 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.735197 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.735201 | controller | - ' exit 1' 2026-03-20 16:26:25.735205 | controller | - ' fi' 2026-03-20 16:26:25.735209 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.735213 | controller | - ' ' 2026-03-20 16:26:25.735217 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.735222 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.735226 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.735230 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.735234 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.735238 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.735243 | controller | -- \' 2026-03-20 16:26:25.735247 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.735251 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.735255 | controller | - ' logger.go:42: 16:11:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.735259 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.735264 | controller | - ' logger.go:42: 16:11:22 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.735268 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.735272 | controller | found' 2026-03-20 16:26:25.735276 | controller | - ' logger.go:42: 16:11:23 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.735280 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.735284 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.735289 | controller | - ' ' 2026-03-20 16:26:25.735293 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.735297 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.735301 | controller | - ' ' 2026-03-20 16:26:25.735306 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.735310 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.735314 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.735318 | controller | - ' exit 1' 2026-03-20 16:26:25.735322 | controller | - ' fi' 2026-03-20 16:26:25.735327 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.735331 | controller | - ' ' 2026-03-20 16:26:25.735335 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.735339 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.735344 | controller | | base64 -d)' 2026-03-20 16:26:25.735348 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.735352 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.735356 | controller | - ' exit 1' 2026-03-20 16:26:25.735360 | controller | - ' fi' 2026-03-20 16:26:25.735367 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.735371 | controller | - ' ' 2026-03-20 16:26:25.735375 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.735379 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.735383 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.735387 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.735392 | controller | --timeout=300s' 2026-03-20 16:26:25.735396 | controller | - ' ' 2026-03-20 16:26:25.735400 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.735411 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.735416 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.735420 | controller | - ' ' 2026-03-20 16:26:25.735424 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.735428 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.735433 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.735437 | controller | - ' ' 2026-03-20 16:26:25.735441 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.735445 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.735450 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.735454 | controller | available"' 2026-03-20 16:26:25.735458 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.735462 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.735466 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.735471 | controller | - ' exit 1' 2026-03-20 16:26:25.735475 | controller | - ' fi' 2026-03-20 16:26:25.735479 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.735483 | controller | - ' ' 2026-03-20 16:26:25.735487 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.735492 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.735496 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.735500 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.735504 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.735508 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.735515 | controller | -- \' 2026-03-20 16:26:25.735519 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.735524 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.735528 | controller | - ' logger.go:42: 16:11:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.735533 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.735537 | controller | - ' logger.go:42: 16:11:23 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.735541 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.735548 | controller | found' 2026-03-20 16:26:25.735552 | controller | - ' logger.go:42: 16:11:24 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.735557 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.735561 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.735565 | controller | - ' ' 2026-03-20 16:26:25.735569 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.735573 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.735578 | controller | - ' ' 2026-03-20 16:26:25.735582 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.735586 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.735590 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.735595 | controller | - ' exit 1' 2026-03-20 16:26:25.735599 | controller | - ' fi' 2026-03-20 16:26:25.735603 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.735607 | controller | - ' ' 2026-03-20 16:26:25.735613 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.735617 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.735621 | controller | | base64 -d)' 2026-03-20 16:26:25.735626 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.735630 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.735634 | controller | - ' exit 1' 2026-03-20 16:26:25.735661 | controller | - ' fi' 2026-03-20 16:26:25.735666 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.735670 | controller | - ' ' 2026-03-20 16:26:25.735674 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.735679 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.735683 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.735687 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.735691 | controller | --timeout=300s' 2026-03-20 16:26:25.735705 | controller | - ' ' 2026-03-20 16:26:25.735710 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.735714 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.735725 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.735730 | controller | - ' ' 2026-03-20 16:26:25.735734 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.735738 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.735743 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.735747 | controller | - ' ' 2026-03-20 16:26:25.735751 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.735755 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.735759 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.735764 | controller | available"' 2026-03-20 16:26:25.735768 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.735772 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.735776 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.735783 | controller | - ' exit 1' 2026-03-20 16:26:25.735787 | controller | - ' fi' 2026-03-20 16:26:25.735791 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.735795 | controller | - ' ' 2026-03-20 16:26:25.735799 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.735804 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.735808 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.735812 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.735816 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.735820 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.735824 | controller | -- \' 2026-03-20 16:26:25.735829 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.735833 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.735837 | controller | - ' logger.go:42: 16:11:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.735841 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.735845 | controller | - ' logger.go:42: 16:11:25 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.735850 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.735854 | controller | found' 2026-03-20 16:26:25.735858 | controller | - ' logger.go:42: 16:11:26 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.735862 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.735866 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.735870 | controller | - ' ' 2026-03-20 16:26:25.735875 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.735879 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.735883 | controller | - ' ' 2026-03-20 16:26:25.735889 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.735894 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.735898 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.735903 | controller | - ' exit 1' 2026-03-20 16:26:25.735908 | controller | - ' fi' 2026-03-20 16:26:25.735912 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.735916 | controller | - ' ' 2026-03-20 16:26:25.735920 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.735924 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.735929 | controller | | base64 -d)' 2026-03-20 16:26:25.735933 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.735937 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.735941 | controller | - ' exit 1' 2026-03-20 16:26:25.735946 | controller | - ' fi' 2026-03-20 16:26:25.735950 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.735954 | controller | - ' ' 2026-03-20 16:26:25.735958 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.735962 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.735967 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.735971 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.735987 | controller | --timeout=300s' 2026-03-20 16:26:25.735991 | controller | - ' ' 2026-03-20 16:26:25.735995 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.736000 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.736011 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.736016 | controller | - ' ' 2026-03-20 16:26:25.736020 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.736024 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.736028 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.736032 | controller | - ' ' 2026-03-20 16:26:25.736037 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.736041 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.736045 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.736049 | controller | available"' 2026-03-20 16:26:25.736054 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.736058 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.736062 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.736066 | controller | - ' exit 1' 2026-03-20 16:26:25.736070 | controller | - ' fi' 2026-03-20 16:26:25.736075 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.736080 | controller | - ' ' 2026-03-20 16:26:25.736085 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.736091 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.736096 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.736102 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.736107 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.736113 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.736117 | controller | -- \' 2026-03-20 16:26:25.736122 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.736126 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.736130 | controller | - ' logger.go:42: 16:11:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.736134 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.736141 | controller | - ' logger.go:42: 16:11:26 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.736145 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.736149 | controller | found' 2026-03-20 16:26:25.736153 | controller | - ' logger.go:42: 16:11:27 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.736158 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.736162 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.736166 | controller | - ' ' 2026-03-20 16:26:25.736170 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.736174 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.736180 | controller | - ' ' 2026-03-20 16:26:25.736185 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.736189 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.736193 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.736197 | controller | - ' exit 1' 2026-03-20 16:26:25.736202 | controller | - ' fi' 2026-03-20 16:26:25.736206 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.736210 | controller | - ' ' 2026-03-20 16:26:25.736214 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.736218 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.736223 | controller | | base64 -d)' 2026-03-20 16:26:25.736227 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.736231 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.736235 | controller | - ' exit 1' 2026-03-20 16:26:25.736239 | controller | - ' fi' 2026-03-20 16:26:25.736244 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.736250 | controller | - ' ' 2026-03-20 16:26:25.736256 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.736261 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.736267 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.736273 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.736278 | controller | --timeout=300s' 2026-03-20 16:26:25.736284 | controller | - ' ' 2026-03-20 16:26:25.736289 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.736294 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.736306 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.736311 | controller | - ' ' 2026-03-20 16:26:25.736315 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.736319 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.736324 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.736328 | controller | - ' ' 2026-03-20 16:26:25.736332 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.736336 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.736340 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.736344 | controller | available"' 2026-03-20 16:26:25.736349 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.736353 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.736357 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.736361 | controller | - ' exit 1' 2026-03-20 16:26:25.736366 | controller | - ' fi' 2026-03-20 16:26:25.736372 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.736378 | controller | - ' ' 2026-03-20 16:26:25.736384 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.736389 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.736395 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.736403 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.736407 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.736411 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.736415 | controller | -- \' 2026-03-20 16:26:25.736420 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.736424 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.736428 | controller | - ' logger.go:42: 16:11:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.736432 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.736436 | controller | - ' logger.go:42: 16:11:27 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.736441 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.736445 | controller | found' 2026-03-20 16:26:25.736449 | controller | - ' logger.go:42: 16:11:28 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.736453 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.736457 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.736462 | controller | - ' ' 2026-03-20 16:26:25.736466 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.736470 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.736474 | controller | - ' ' 2026-03-20 16:26:25.736478 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.736482 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.736487 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.736491 | controller | - ' exit 1' 2026-03-20 16:26:25.736495 | controller | - ' fi' 2026-03-20 16:26:25.736499 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.736504 | controller | - ' ' 2026-03-20 16:26:25.736508 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.736512 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.736516 | controller | | base64 -d)' 2026-03-20 16:26:25.736520 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.736524 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.736528 | controller | - ' exit 1' 2026-03-20 16:26:25.736533 | controller | - ' fi' 2026-03-20 16:26:25.736537 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.736541 | controller | - ' ' 2026-03-20 16:26:25.736545 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.736550 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.736554 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.736558 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.736562 | controller | --timeout=300s' 2026-03-20 16:26:25.736566 | controller | - ' ' 2026-03-20 16:26:25.736570 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.736574 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.736579 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.736583 | controller | - ' ' 2026-03-20 16:26:25.736597 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.736602 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.736610 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.736616 | controller | - ' ' 2026-03-20 16:26:25.736621 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.736627 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.736632 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.736636 | controller | available"' 2026-03-20 16:26:25.736640 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.736644 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.736648 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.736653 | controller | - ' exit 1' 2026-03-20 16:26:25.736657 | controller | - ' fi' 2026-03-20 16:26:25.736661 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.736665 | controller | - ' ' 2026-03-20 16:26:25.736669 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.736673 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.736678 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.736682 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.736686 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.736690 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.736710 | controller | -- \' 2026-03-20 16:26:25.736714 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.736718 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.736722 | controller | - ' logger.go:42: 16:11:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.736727 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.736731 | controller | - ' logger.go:42: 16:11:28 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.736737 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.736742 | controller | found' 2026-03-20 16:26:25.736746 | controller | - ' logger.go:42: 16:11:29 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.736750 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.736754 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.736758 | controller | - ' ' 2026-03-20 16:26:25.736763 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.736767 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.736771 | controller | - ' ' 2026-03-20 16:26:25.736775 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.736779 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.736783 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.736788 | controller | - ' exit 1' 2026-03-20 16:26:25.736792 | controller | - ' fi' 2026-03-20 16:26:25.736796 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.736800 | controller | - ' ' 2026-03-20 16:26:25.736804 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.736811 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.736815 | controller | | base64 -d)' 2026-03-20 16:26:25.736819 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.736824 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.736828 | controller | - ' exit 1' 2026-03-20 16:26:25.736832 | controller | - ' fi' 2026-03-20 16:26:25.736836 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.736840 | controller | - ' ' 2026-03-20 16:26:25.736845 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.736849 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.736853 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.736857 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.736861 | controller | --timeout=300s' 2026-03-20 16:26:25.736866 | controller | - ' ' 2026-03-20 16:26:25.736870 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.736874 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.736878 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.736882 | controller | - ' ' 2026-03-20 16:26:25.736894 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.736899 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.736903 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.736907 | controller | - ' ' 2026-03-20 16:26:25.736912 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.736916 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.736920 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.736924 | controller | available"' 2026-03-20 16:26:25.736931 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.736935 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.736940 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.736945 | controller | - ' exit 1' 2026-03-20 16:26:25.736949 | controller | - ' fi' 2026-03-20 16:26:25.736953 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.736957 | controller | - ' ' 2026-03-20 16:26:25.736961 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.736966 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.736970 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.736974 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.736978 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.736982 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.736986 | controller | -- \' 2026-03-20 16:26:25.736991 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.736998 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.737002 | controller | - ' logger.go:42: 16:11:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.737006 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.737010 | controller | - ' logger.go:42: 16:11:29 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.737014 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.737019 | controller | found' 2026-03-20 16:26:25.737023 | controller | - ' logger.go:42: 16:11:30 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.737027 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.737031 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.737035 | controller | - ' ' 2026-03-20 16:26:25.737039 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.737044 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.737048 | controller | - ' ' 2026-03-20 16:26:25.737052 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.737056 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.737060 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.737065 | controller | - ' exit 1' 2026-03-20 16:26:25.737069 | controller | - ' fi' 2026-03-20 16:26:25.737073 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.737077 | controller | - ' ' 2026-03-20 16:26:25.737082 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.737086 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.737090 | controller | | base64 -d)' 2026-03-20 16:26:25.737094 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.737098 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.737103 | controller | - ' exit 1' 2026-03-20 16:26:25.737107 | controller | - ' fi' 2026-03-20 16:26:25.737111 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.737115 | controller | - ' ' 2026-03-20 16:26:25.737119 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.737123 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.737128 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.737132 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.737136 | controller | --timeout=300s' 2026-03-20 16:26:25.737140 | controller | - ' ' 2026-03-20 16:26:25.737144 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.737148 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.737153 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.737157 | controller | - ' ' 2026-03-20 16:26:25.737161 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.737173 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.737177 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.737182 | controller | - ' ' 2026-03-20 16:26:25.737186 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.737190 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.737199 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.737203 | controller | available"' 2026-03-20 16:26:25.737207 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.737211 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.737216 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.737220 | controller | - ' exit 1' 2026-03-20 16:26:25.737224 | controller | - ' fi' 2026-03-20 16:26:25.737230 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.737234 | controller | - ' ' 2026-03-20 16:26:25.737238 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.737242 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.737247 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.737251 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.737255 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.737259 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.737263 | controller | -- \' 2026-03-20 16:26:25.737267 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.737272 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.737276 | controller | - ' logger.go:42: 16:11:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.737280 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.737284 | controller | - ' logger.go:42: 16:11:30 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.737288 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.737292 | controller | found' 2026-03-20 16:26:25.737298 | controller | - ' logger.go:42: 16:11:31 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.737302 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.737307 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.737311 | controller | - ' ' 2026-03-20 16:26:25.737315 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.737319 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.737323 | controller | - ' ' 2026-03-20 16:26:25.737327 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.737331 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.737336 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.737340 | controller | - ' exit 1' 2026-03-20 16:26:25.737344 | controller | - ' fi' 2026-03-20 16:26:25.737348 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.737352 | controller | - ' ' 2026-03-20 16:26:25.737356 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.737361 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.737365 | controller | | base64 -d)' 2026-03-20 16:26:25.737369 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.737373 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.737377 | controller | - ' exit 1' 2026-03-20 16:26:25.737382 | controller | - ' fi' 2026-03-20 16:26:25.737386 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.737392 | controller | - ' ' 2026-03-20 16:26:25.737396 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.737400 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.737404 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.737408 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.737413 | controller | --timeout=300s' 2026-03-20 16:26:25.737417 | controller | - ' ' 2026-03-20 16:26:25.737421 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.737425 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.737429 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.737434 | controller | - ' ' 2026-03-20 16:26:25.737438 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.737449 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.737453 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.737458 | controller | - ' ' 2026-03-20 16:26:25.737462 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.737466 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.737470 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.737474 | controller | available"' 2026-03-20 16:26:25.737479 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.737483 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.737487 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.737491 | controller | - ' exit 1' 2026-03-20 16:26:25.737496 | controller | - ' fi' 2026-03-20 16:26:25.737500 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.737504 | controller | - ' ' 2026-03-20 16:26:25.737508 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.737512 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.737517 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.737521 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.737525 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.737529 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.737534 | controller | -- \' 2026-03-20 16:26:25.737538 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.737542 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.737548 | controller | - ' logger.go:42: 16:11:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.737552 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.737556 | controller | - ' logger.go:42: 16:11:32 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.737561 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.737565 | controller | found' 2026-03-20 16:26:25.737571 | controller | - ' logger.go:42: 16:11:33 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.737575 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.737579 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.737583 | controller | - ' ' 2026-03-20 16:26:25.737588 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.737592 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.737596 | controller | - ' ' 2026-03-20 16:26:25.737600 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.737604 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.737609 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.737613 | controller | - ' exit 1' 2026-03-20 16:26:25.737617 | controller | - ' fi' 2026-03-20 16:26:25.737621 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.737625 | controller | - ' ' 2026-03-20 16:26:25.737630 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.737634 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.737638 | controller | | base64 -d)' 2026-03-20 16:26:25.737642 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.737646 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.737651 | controller | - ' exit 1' 2026-03-20 16:26:25.737655 | controller | - ' fi' 2026-03-20 16:26:25.737659 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.737663 | controller | - ' ' 2026-03-20 16:26:25.737667 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.737672 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.737676 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.737680 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.737684 | controller | --timeout=300s' 2026-03-20 16:26:25.737688 | controller | - ' ' 2026-03-20 16:26:25.737703 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.737709 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.737714 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.737718 | controller | - ' ' 2026-03-20 16:26:25.737722 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.737726 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.737737 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.737742 | controller | - ' ' 2026-03-20 16:26:25.737746 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.737750 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.737755 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.737759 | controller | available"' 2026-03-20 16:26:25.737763 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.737767 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.737771 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.737776 | controller | - ' exit 1' 2026-03-20 16:26:25.737782 | controller | - ' fi' 2026-03-20 16:26:25.737786 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.737790 | controller | - ' ' 2026-03-20 16:26:25.737795 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.737799 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.737803 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.737807 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.737811 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.737815 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.737820 | controller | -- \' 2026-03-20 16:26:25.737826 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.737830 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.737835 | controller | - ' logger.go:42: 16:11:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.737839 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.737843 | controller | - ' logger.go:42: 16:11:33 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.737847 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.737851 | controller | found' 2026-03-20 16:26:25.737856 | controller | - ' logger.go:42: 16:11:34 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.737860 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.737864 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.737871 | controller | - ' ' 2026-03-20 16:26:25.737875 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.737881 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.737885 | controller | - ' ' 2026-03-20 16:26:25.737890 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.737894 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.737898 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.737903 | controller | - ' exit 1' 2026-03-20 16:26:25.737907 | controller | - ' fi' 2026-03-20 16:26:25.737911 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.737915 | controller | - ' ' 2026-03-20 16:26:25.737920 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.737924 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.737928 | controller | | base64 -d)' 2026-03-20 16:26:25.737932 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.737936 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.737940 | controller | - ' exit 1' 2026-03-20 16:26:25.737945 | controller | - ' fi' 2026-03-20 16:26:25.737949 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.737953 | controller | - ' ' 2026-03-20 16:26:25.737959 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.737963 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.737968 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.737972 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.737976 | controller | --timeout=300s' 2026-03-20 16:26:25.737983 | controller | - ' ' 2026-03-20 16:26:25.737987 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.737992 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.737996 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.738000 | controller | - ' ' 2026-03-20 16:26:25.738005 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.738009 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.738020 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.738025 | controller | - ' ' 2026-03-20 16:26:25.738029 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.738033 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.738038 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.738042 | controller | available"' 2026-03-20 16:26:25.738046 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.738050 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.738054 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.738059 | controller | - ' exit 1' 2026-03-20 16:26:25.738063 | controller | - ' fi' 2026-03-20 16:26:25.738067 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.738071 | controller | - ' ' 2026-03-20 16:26:25.738075 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.738080 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.738084 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.738088 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.738092 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.738096 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.738101 | controller | -- \' 2026-03-20 16:26:25.738105 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.738109 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.738113 | controller | - ' logger.go:42: 16:11:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.738118 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.738122 | controller | - ' logger.go:42: 16:11:34 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.738126 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.738130 | controller | found' 2026-03-20 16:26:25.738136 | controller | - ' logger.go:42: 16:11:35 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.738140 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.738144 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.738149 | controller | - ' ' 2026-03-20 16:26:25.738153 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.738157 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.738161 | controller | - ' ' 2026-03-20 16:26:25.738165 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.738171 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.738176 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.738180 | controller | - ' exit 1' 2026-03-20 16:26:25.738184 | controller | - ' fi' 2026-03-20 16:26:25.738189 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.738193 | controller | - ' ' 2026-03-20 16:26:25.738199 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.738203 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.738207 | controller | | base64 -d)' 2026-03-20 16:26:25.738211 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.738215 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.738219 | controller | - ' exit 1' 2026-03-20 16:26:25.738224 | controller | - ' fi' 2026-03-20 16:26:25.738228 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.738232 | controller | - ' ' 2026-03-20 16:26:25.738236 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.738240 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.738244 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.738249 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.738253 | controller | --timeout=300s' 2026-03-20 16:26:25.738257 | controller | - ' ' 2026-03-20 16:26:25.738261 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.738265 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.738269 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.738273 | controller | - ' ' 2026-03-20 16:26:25.738278 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.738282 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.738286 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.738298 | controller | - ' ' 2026-03-20 16:26:25.738302 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.738306 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.738311 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.738315 | controller | available"' 2026-03-20 16:26:25.738319 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.738323 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.738327 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.738331 | controller | - ' exit 1' 2026-03-20 16:26:25.738335 | controller | - ' fi' 2026-03-20 16:26:25.738340 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.738344 | controller | - ' ' 2026-03-20 16:26:25.738348 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.738352 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.738356 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.738361 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.738368 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.738372 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.738376 | controller | -- \' 2026-03-20 16:26:25.738380 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.738385 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.738389 | controller | - ' logger.go:42: 16:11:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.738393 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.738397 | controller | - ' logger.go:42: 16:11:35 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.738401 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.738405 | controller | found' 2026-03-20 16:26:25.738410 | controller | - ' logger.go:42: 16:11:36 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.738414 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.738418 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.738422 | controller | - ' ' 2026-03-20 16:26:25.738427 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.738433 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.738437 | controller | - ' ' 2026-03-20 16:26:25.738441 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.738445 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.738449 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.738454 | controller | - ' exit 1' 2026-03-20 16:26:25.738458 | controller | - ' fi' 2026-03-20 16:26:25.738462 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.738466 | controller | - ' ' 2026-03-20 16:26:25.738470 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.738475 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.738479 | controller | | base64 -d)' 2026-03-20 16:26:25.738483 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.738487 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.738491 | controller | - ' exit 1' 2026-03-20 16:26:25.738495 | controller | - ' fi' 2026-03-20 16:26:25.738500 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.738506 | controller | - ' ' 2026-03-20 16:26:25.738510 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.738514 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.738518 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.738522 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.738527 | controller | --timeout=300s' 2026-03-20 16:26:25.738531 | controller | - ' ' 2026-03-20 16:26:25.738535 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.738539 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.738543 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.738548 | controller | - ' ' 2026-03-20 16:26:25.738552 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.738558 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.738562 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.738566 | controller | - ' ' 2026-03-20 16:26:25.738577 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.738584 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.738588 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.738592 | controller | available"' 2026-03-20 16:26:25.738596 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.738601 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.738615 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.738619 | controller | - ' exit 1' 2026-03-20 16:26:25.738623 | controller | - ' fi' 2026-03-20 16:26:25.738627 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.738632 | controller | - ' ' 2026-03-20 16:26:25.738636 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.738640 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.738644 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.738648 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.738652 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.738657 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.738661 | controller | -- \' 2026-03-20 16:26:25.738665 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.738669 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.738673 | controller | - ' logger.go:42: 16:11:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.738678 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.738690 | controller | - ' logger.go:42: 16:11:36 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.738706 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.738710 | controller | found' 2026-03-20 16:26:25.738715 | controller | - ' logger.go:42: 16:11:37 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.738719 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.738723 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.738727 | controller | - ' ' 2026-03-20 16:26:25.738732 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.738736 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.738740 | controller | - ' ' 2026-03-20 16:26:25.738744 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.738748 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.738753 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.738757 | controller | - ' exit 1' 2026-03-20 16:26:25.738761 | controller | - ' fi' 2026-03-20 16:26:25.738765 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.738769 | controller | - ' ' 2026-03-20 16:26:25.738773 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.738780 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.738792 | controller | | base64 -d)' 2026-03-20 16:26:25.738796 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.738800 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.738804 | controller | - ' exit 1' 2026-03-20 16:26:25.738808 | controller | - ' fi' 2026-03-20 16:26:25.738813 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.738817 | controller | - ' ' 2026-03-20 16:26:25.738821 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.738825 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.738830 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.738834 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.738838 | controller | --timeout=300s' 2026-03-20 16:26:25.738844 | controller | - ' ' 2026-03-20 16:26:25.738849 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.738853 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.738857 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.738861 | controller | - ' ' 2026-03-20 16:26:25.738873 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.738877 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.738882 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.738886 | controller | - ' ' 2026-03-20 16:26:25.738890 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.738902 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.738906 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.738911 | controller | available"' 2026-03-20 16:26:25.738915 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.738919 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.738923 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.738927 | controller | - ' exit 1' 2026-03-20 16:26:25.738932 | controller | - ' fi' 2026-03-20 16:26:25.738936 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.738940 | controller | - ' ' 2026-03-20 16:26:25.738944 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.738948 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.738953 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.738957 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.738961 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.738965 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.738969 | controller | -- \' 2026-03-20 16:26:25.738973 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.738978 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.738984 | controller | - ' logger.go:42: 16:11:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.738988 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.738992 | controller | - ' logger.go:42: 16:11:37 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.738996 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.739001 | controller | found' 2026-03-20 16:26:25.739005 | controller | - ' logger.go:42: 16:11:38 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.739009 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.739013 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.739017 | controller | - ' ' 2026-03-20 16:26:25.739022 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.739026 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.739030 | controller | - ' ' 2026-03-20 16:26:25.739034 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.739038 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.739043 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.739047 | controller | - ' exit 1' 2026-03-20 16:26:25.739053 | controller | - ' fi' 2026-03-20 16:26:25.739059 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.739063 | controller | - ' ' 2026-03-20 16:26:25.739067 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.739071 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.739076 | controller | | base64 -d)' 2026-03-20 16:26:25.739080 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.739084 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.739088 | controller | - ' exit 1' 2026-03-20 16:26:25.739093 | controller | - ' fi' 2026-03-20 16:26:25.739097 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.739101 | controller | - ' ' 2026-03-20 16:26:25.739105 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.739109 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.739114 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.739118 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.739122 | controller | --timeout=300s' 2026-03-20 16:26:25.739126 | controller | - ' ' 2026-03-20 16:26:25.739130 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.739135 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.739139 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.739143 | controller | - ' ' 2026-03-20 16:26:25.739147 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.739151 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.739155 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.739160 | controller | - ' ' 2026-03-20 16:26:25.739164 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.739175 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.739182 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.739186 | controller | available"' 2026-03-20 16:26:25.739190 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.739194 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.739199 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.739203 | controller | - ' exit 1' 2026-03-20 16:26:25.739207 | controller | - ' fi' 2026-03-20 16:26:25.739211 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.739215 | controller | - ' ' 2026-03-20 16:26:25.739220 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.739224 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.739228 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.739232 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.739236 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.739241 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.739245 | controller | -- \' 2026-03-20 16:26:25.739249 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.739253 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.739257 | controller | - ' logger.go:42: 16:11:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.739262 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.739266 | controller | - ' logger.go:42: 16:11:38 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.739270 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.739274 | controller | found' 2026-03-20 16:26:25.739278 | controller | - ' logger.go:42: 16:11:39 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.739283 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.739287 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.739291 | controller | - ' ' 2026-03-20 16:26:25.739295 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.739299 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.739304 | controller | - ' ' 2026-03-20 16:26:25.739308 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.739312 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.739316 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.739320 | controller | - ' exit 1' 2026-03-20 16:26:25.739325 | controller | - ' fi' 2026-03-20 16:26:25.739329 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.739333 | controller | - ' ' 2026-03-20 16:26:25.739337 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.739341 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.739346 | controller | | base64 -d)' 2026-03-20 16:26:25.739350 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.739354 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.739358 | controller | - ' exit 1' 2026-03-20 16:26:25.739362 | controller | - ' fi' 2026-03-20 16:26:25.739367 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.739371 | controller | - ' ' 2026-03-20 16:26:25.739377 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.739381 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.739386 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.739390 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.739394 | controller | --timeout=300s' 2026-03-20 16:26:25.739398 | controller | - ' ' 2026-03-20 16:26:25.739402 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.739406 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.739411 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.739415 | controller | - ' ' 2026-03-20 16:26:25.739419 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.739423 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.739428 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.739432 | controller | - ' ' 2026-03-20 16:26:25.739436 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.739440 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.739451 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.739455 | controller | available"' 2026-03-20 16:26:25.739459 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.739466 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.739470 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.739474 | controller | - ' exit 1' 2026-03-20 16:26:25.739478 | controller | - ' fi' 2026-03-20 16:26:25.739482 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.739487 | controller | - ' ' 2026-03-20 16:26:25.739491 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.739495 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.739499 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.739503 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.739507 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.739511 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.739516 | controller | -- \' 2026-03-20 16:26:25.739520 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.739524 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.739528 | controller | - ' logger.go:42: 16:11:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.739532 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.739537 | controller | - ' logger.go:42: 16:11:40 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.739541 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.739545 | controller | found' 2026-03-20 16:26:25.739551 | controller | - ' logger.go:42: 16:11:41 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.739557 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.739561 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.739565 | controller | - ' ' 2026-03-20 16:26:25.739569 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.739574 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.739578 | controller | - ' ' 2026-03-20 16:26:25.739582 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.739586 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.739590 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.739595 | controller | - ' exit 1' 2026-03-20 16:26:25.739599 | controller | - ' fi' 2026-03-20 16:26:25.739603 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.739610 | controller | - ' ' 2026-03-20 16:26:25.739615 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.739619 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.739623 | controller | | base64 -d)' 2026-03-20 16:26:25.739627 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.739631 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.739636 | controller | - ' exit 1' 2026-03-20 16:26:25.739666 | controller | - ' fi' 2026-03-20 16:26:25.739672 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.739677 | controller | - ' ' 2026-03-20 16:26:25.739682 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.739687 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.739704 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.739712 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.739717 | controller | --timeout=300s' 2026-03-20 16:26:25.739722 | controller | - ' ' 2026-03-20 16:26:25.739727 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.739732 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.739744 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.739749 | controller | - ' ' 2026-03-20 16:26:25.739754 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.739759 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.739764 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.739769 | controller | - ' ' 2026-03-20 16:26:25.739774 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.739779 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.739783 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.739800 | controller | available"' 2026-03-20 16:26:25.739805 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.739810 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.739815 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.739820 | controller | - ' exit 1' 2026-03-20 16:26:25.739825 | controller | - ' fi' 2026-03-20 16:26:25.739830 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.739838 | controller | - ' ' 2026-03-20 16:26:25.739842 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.739847 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.739852 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.739857 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.739862 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.739867 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.739872 | controller | -- \' 2026-03-20 16:26:25.739877 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.739882 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.739887 | controller | - ' logger.go:42: 16:11:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.739892 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.739897 | controller | - ' logger.go:42: 16:11:41 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.739902 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.739906 | controller | found' 2026-03-20 16:26:25.739910 | controller | - ' logger.go:42: 16:11:42 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.739914 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.739919 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.739923 | controller | - ' ' 2026-03-20 16:26:25.739927 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.739931 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.739936 | controller | - ' ' 2026-03-20 16:26:25.739940 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.739944 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.739948 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.739955 | controller | - ' exit 1' 2026-03-20 16:26:25.739959 | controller | - ' fi' 2026-03-20 16:26:25.739963 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.739967 | controller | - ' ' 2026-03-20 16:26:25.739972 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.739976 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.739980 | controller | | base64 -d)' 2026-03-20 16:26:25.739984 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.739988 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.739992 | controller | - ' exit 1' 2026-03-20 16:26:25.739997 | controller | - ' fi' 2026-03-20 16:26:25.740001 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.740005 | controller | - ' ' 2026-03-20 16:26:25.740009 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.740013 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.740018 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.740022 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.740026 | controller | --timeout=300s' 2026-03-20 16:26:25.740030 | controller | - ' ' 2026-03-20 16:26:25.740035 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.740041 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.740045 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.740049 | controller | - ' ' 2026-03-20 16:26:25.740053 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.740058 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.740062 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.740066 | controller | - ' ' 2026-03-20 16:26:25.740070 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.740074 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.740079 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.740083 | controller | available"' 2026-03-20 16:26:25.740095 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.740099 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.740104 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.740108 | controller | - ' exit 1' 2026-03-20 16:26:25.740112 | controller | - ' fi' 2026-03-20 16:26:25.740118 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.740122 | controller | - ' ' 2026-03-20 16:26:25.740127 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.740131 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.740135 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.740139 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.740143 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.740148 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.740152 | controller | -- \' 2026-03-20 16:26:25.740156 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.740160 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.740164 | controller | - ' logger.go:42: 16:11:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.740168 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.740173 | controller | - ' logger.go:42: 16:11:42 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.740177 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.740181 | controller | found' 2026-03-20 16:26:25.740185 | controller | - ' logger.go:42: 16:11:43 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.740189 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.740194 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.740198 | controller | - ' ' 2026-03-20 16:26:25.740202 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.740206 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.740210 | controller | - ' ' 2026-03-20 16:26:25.740214 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.740221 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.740225 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.740229 | controller | - ' exit 1' 2026-03-20 16:26:25.740233 | controller | - ' fi' 2026-03-20 16:26:25.740238 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.740242 | controller | - ' ' 2026-03-20 16:26:25.740246 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.740250 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.740254 | controller | | base64 -d)' 2026-03-20 16:26:25.740259 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.740263 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.740267 | controller | - ' exit 1' 2026-03-20 16:26:25.740271 | controller | - ' fi' 2026-03-20 16:26:25.740275 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.740279 | controller | - ' ' 2026-03-20 16:26:25.740283 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.740288 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.740293 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.740298 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.740302 | controller | --timeout=300s' 2026-03-20 16:26:25.740306 | controller | - ' ' 2026-03-20 16:26:25.740310 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.740314 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.740319 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.740323 | controller | - ' ' 2026-03-20 16:26:25.740327 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.740331 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.740335 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.740340 | controller | - ' ' 2026-03-20 16:26:25.740344 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.740348 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.740352 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.740356 | controller | available"' 2026-03-20 16:26:25.740361 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.740373 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.740378 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.740382 | controller | - ' exit 1' 2026-03-20 16:26:25.740386 | controller | - ' fi' 2026-03-20 16:26:25.740390 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.740394 | controller | - ' ' 2026-03-20 16:26:25.740399 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.740403 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.740407 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.740411 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.740418 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.740422 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.740426 | controller | -- \' 2026-03-20 16:26:25.740431 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.740435 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.740439 | controller | - ' logger.go:42: 16:11:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.740443 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.740447 | controller | - ' logger.go:42: 16:11:43 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.740451 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.740456 | controller | found' 2026-03-20 16:26:25.740460 | controller | - ' logger.go:42: 16:11:44 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.740464 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.740468 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.740472 | controller | - ' ' 2026-03-20 16:26:25.740477 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.740481 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.740485 | controller | - ' ' 2026-03-20 16:26:25.740489 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.740493 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.740498 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.740502 | controller | - ' exit 1' 2026-03-20 16:26:25.740506 | controller | - ' fi' 2026-03-20 16:26:25.740510 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.740514 | controller | - ' ' 2026-03-20 16:26:25.740519 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.740523 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.740527 | controller | | base64 -d)' 2026-03-20 16:26:25.740531 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.740535 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.740540 | controller | - ' exit 1' 2026-03-20 16:26:25.740544 | controller | - ' fi' 2026-03-20 16:26:25.740548 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.740552 | controller | - ' ' 2026-03-20 16:26:25.740557 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.740561 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.740565 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.740569 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.740573 | controller | --timeout=300s' 2026-03-20 16:26:25.740577 | controller | - ' ' 2026-03-20 16:26:25.740582 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.740586 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.740590 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.740594 | controller | - ' ' 2026-03-20 16:26:25.740598 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.740602 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.740608 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.740613 | controller | - ' ' 2026-03-20 16:26:25.740617 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.740621 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.740625 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.740629 | controller | available"' 2026-03-20 16:26:25.740633 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.740644 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.740649 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.740653 | controller | - ' exit 1' 2026-03-20 16:26:25.740657 | controller | - ' fi' 2026-03-20 16:26:25.740661 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.740666 | controller | - ' ' 2026-03-20 16:26:25.740670 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.740674 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.740678 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.740683 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.740716 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.740721 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.740725 | controller | -- \' 2026-03-20 16:26:25.740730 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.740734 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.740738 | controller | - ' logger.go:42: 16:11:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.740742 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.740746 | controller | - ' logger.go:42: 16:11:44 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.740751 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.740755 | controller | found' 2026-03-20 16:26:25.740759 | controller | - ' logger.go:42: 16:11:45 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.740763 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.740767 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.740771 | controller | - ' ' 2026-03-20 16:26:25.740776 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.740780 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.740784 | controller | - ' ' 2026-03-20 16:26:25.740788 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.740792 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.740797 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.740801 | controller | - ' exit 1' 2026-03-20 16:26:25.740805 | controller | - ' fi' 2026-03-20 16:26:25.740809 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.740813 | controller | - ' ' 2026-03-20 16:26:25.740819 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.740824 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.740830 | controller | | base64 -d)' 2026-03-20 16:26:25.740834 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.740838 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.740842 | controller | - ' exit 1' 2026-03-20 16:26:25.740847 | controller | - ' fi' 2026-03-20 16:26:25.740851 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.740855 | controller | - ' ' 2026-03-20 16:26:25.740859 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.740863 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.740867 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.740873 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.740877 | controller | --timeout=300s' 2026-03-20 16:26:25.740882 | controller | - ' ' 2026-03-20 16:26:25.740886 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.740890 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.740894 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.740898 | controller | - ' ' 2026-03-20 16:26:25.740902 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.740906 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.740911 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.740915 | controller | - ' ' 2026-03-20 16:26:25.740919 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.740923 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.740927 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.740932 | controller | available"' 2026-03-20 16:26:25.740936 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.740940 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.740952 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.740956 | controller | - ' exit 1' 2026-03-20 16:26:25.740961 | controller | - ' fi' 2026-03-20 16:26:25.740965 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.740969 | controller | - ' ' 2026-03-20 16:26:25.740973 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.740977 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.740981 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.740986 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.740990 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.740994 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.740998 | controller | -- \' 2026-03-20 16:26:25.741002 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.741007 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.741013 | controller | - ' logger.go:42: 16:11:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.741017 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.741021 | controller | - ' logger.go:42: 16:11:45 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.741025 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.741029 | controller | found' 2026-03-20 16:26:25.741034 | controller | - ' logger.go:42: 16:11:46 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.741038 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.741042 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.741046 | controller | - ' ' 2026-03-20 16:26:25.741050 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.741055 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.741059 | controller | - ' ' 2026-03-20 16:26:25.741063 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.741067 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.741071 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.741076 | controller | - ' exit 1' 2026-03-20 16:26:25.741080 | controller | - ' fi' 2026-03-20 16:26:25.741084 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.741088 | controller | - ' ' 2026-03-20 16:26:25.741092 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.741096 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.741101 | controller | | base64 -d)' 2026-03-20 16:26:25.741105 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.741109 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.741113 | controller | - ' exit 1' 2026-03-20 16:26:25.741117 | controller | - ' fi' 2026-03-20 16:26:25.741122 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.741126 | controller | - ' ' 2026-03-20 16:26:25.741130 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.741134 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.741138 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.741143 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.741147 | controller | --timeout=300s' 2026-03-20 16:26:25.741151 | controller | - ' ' 2026-03-20 16:26:25.741158 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.741163 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.741167 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.741171 | controller | - ' ' 2026-03-20 16:26:25.741175 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.741180 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.741184 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.741188 | controller | - ' ' 2026-03-20 16:26:25.741192 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.741196 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.741202 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.741208 | controller | available"' 2026-03-20 16:26:25.741212 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.741216 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.741221 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.741225 | controller | - ' exit 1' 2026-03-20 16:26:25.741236 | controller | - ' fi' 2026-03-20 16:26:25.741241 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.741245 | controller | - ' ' 2026-03-20 16:26:25.741249 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.741253 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.741257 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.741262 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.741266 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.741270 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.741274 | controller | -- \' 2026-03-20 16:26:25.741278 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.741282 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.741287 | controller | - ' logger.go:42: 16:11:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.741291 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.741295 | controller | - ' logger.go:42: 16:11:47 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.741300 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.741304 | controller | found' 2026-03-20 16:26:25.741308 | controller | - ' logger.go:42: 16:11:48 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.741312 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.741317 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.741322 | controller | - ' ' 2026-03-20 16:26:25.741327 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.741331 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.741336 | controller | - ' ' 2026-03-20 16:26:25.741342 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.741346 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.741351 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.741355 | controller | - ' exit 1' 2026-03-20 16:26:25.741359 | controller | - ' fi' 2026-03-20 16:26:25.741363 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.741367 | controller | - ' ' 2026-03-20 16:26:25.741372 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.741376 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.741380 | controller | | base64 -d)' 2026-03-20 16:26:25.741384 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.741388 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.741393 | controller | - ' exit 1' 2026-03-20 16:26:25.741397 | controller | - ' fi' 2026-03-20 16:26:25.741401 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.741405 | controller | - ' ' 2026-03-20 16:26:25.741409 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.741416 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.741420 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.741424 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.741428 | controller | --timeout=300s' 2026-03-20 16:26:25.741432 | controller | - ' ' 2026-03-20 16:26:25.741437 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.741441 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.741445 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.741449 | controller | - ' ' 2026-03-20 16:26:25.741453 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.741458 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.741462 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.741466 | controller | - ' ' 2026-03-20 16:26:25.741596 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.741601 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.741605 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.741609 | controller | available"' 2026-03-20 16:26:25.741613 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.741617 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.741622 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.741626 | controller | - ' exit 1' 2026-03-20 16:26:25.741630 | controller | - ' fi' 2026-03-20 16:26:25.741641 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.741645 | controller | - ' ' 2026-03-20 16:26:25.741650 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.741654 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.741658 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.741662 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.741666 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.741671 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.741675 | controller | -- \' 2026-03-20 16:26:25.741679 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.741683 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.741687 | controller | - ' logger.go:42: 16:11:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.741703 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.741712 | controller | - ' logger.go:42: 16:11:48 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.741716 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.741721 | controller | found' 2026-03-20 16:26:25.741725 | controller | - ' logger.go:42: 16:11:49 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.741729 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.741735 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.741740 | controller | - ' ' 2026-03-20 16:26:25.741744 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.741748 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.741752 | controller | - ' ' 2026-03-20 16:26:25.741756 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.741760 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.741765 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.741769 | controller | - ' exit 1' 2026-03-20 16:26:25.741773 | controller | - ' fi' 2026-03-20 16:26:25.741777 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.741781 | controller | - ' ' 2026-03-20 16:26:25.741785 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.741790 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.741794 | controller | | base64 -d)' 2026-03-20 16:26:25.741798 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.741802 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.741806 | controller | - ' exit 1' 2026-03-20 16:26:25.741811 | controller | - ' fi' 2026-03-20 16:26:25.741815 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.741819 | controller | - ' ' 2026-03-20 16:26:25.741823 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.741827 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.741832 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.741836 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.741840 | controller | --timeout=300s' 2026-03-20 16:26:25.741844 | controller | - ' ' 2026-03-20 16:26:25.741848 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.741852 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.741857 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.741861 | controller | - ' ' 2026-03-20 16:26:25.741865 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.741871 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.741875 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.741879 | controller | - ' ' 2026-03-20 16:26:25.741883 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.741888 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.741892 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.741896 | controller | available"' 2026-03-20 16:26:25.741900 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.741904 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.741908 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.741913 | controller | - ' exit 1' 2026-03-20 16:26:25.741917 | controller | - ' fi' 2026-03-20 16:26:25.741921 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.741935 | controller | - ' ' 2026-03-20 16:26:25.741939 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.741943 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.741948 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.741952 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.741956 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.741960 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.741965 | controller | -- \' 2026-03-20 16:26:25.741969 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.741973 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.741977 | controller | - ' logger.go:42: 16:11:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.741981 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.741986 | controller | - ' logger.go:42: 16:11:49 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.741990 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.741994 | controller | found' 2026-03-20 16:26:25.741998 | controller | - ' logger.go:42: 16:11:50 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.742002 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.742007 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.742011 | controller | - ' ' 2026-03-20 16:26:25.742015 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.742019 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.742023 | controller | - ' ' 2026-03-20 16:26:25.742028 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.742032 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.742036 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.742040 | controller | - ' exit 1' 2026-03-20 16:26:25.742044 | controller | - ' fi' 2026-03-20 16:26:25.742048 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.742053 | controller | - ' ' 2026-03-20 16:26:25.742057 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.742061 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.742067 | controller | | base64 -d)' 2026-03-20 16:26:25.742071 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.742075 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.742080 | controller | - ' exit 1' 2026-03-20 16:26:25.742084 | controller | - ' fi' 2026-03-20 16:26:25.742088 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.742092 | controller | - ' ' 2026-03-20 16:26:25.742096 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.742101 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.742105 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.742109 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.742113 | controller | --timeout=300s' 2026-03-20 16:26:25.742117 | controller | - ' ' 2026-03-20 16:26:25.742121 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.742127 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.742131 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.742136 | controller | - ' ' 2026-03-20 16:26:25.742140 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.742144 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.742148 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.742152 | controller | - ' ' 2026-03-20 16:26:25.742157 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.742161 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.742166 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.742171 | controller | available"' 2026-03-20 16:26:25.742175 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.742181 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.742185 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.742189 | controller | - ' exit 1' 2026-03-20 16:26:25.742193 | controller | - ' fi' 2026-03-20 16:26:25.742197 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.742202 | controller | - ' ' 2026-03-20 16:26:25.742213 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.742218 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.742222 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.742226 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.742230 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.742234 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.742239 | controller | -- \' 2026-03-20 16:26:25.742243 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.742247 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.742251 | controller | - ' logger.go:42: 16:11:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.742255 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.742260 | controller | - ' logger.go:42: 16:11:51 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.742264 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.742268 | controller | found' 2026-03-20 16:26:25.742272 | controller | - ' logger.go:42: 16:11:52 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.742276 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.742281 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.742285 | controller | - ' ' 2026-03-20 16:26:25.742289 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.742293 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.742297 | controller | - ' ' 2026-03-20 16:26:25.742302 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.742306 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.742312 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.742316 | controller | - ' exit 1' 2026-03-20 16:26:25.742320 | controller | - ' fi' 2026-03-20 16:26:25.742324 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.742329 | controller | - ' ' 2026-03-20 16:26:25.742333 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.742337 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.742341 | controller | | base64 -d)' 2026-03-20 16:26:25.742347 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.742352 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.742356 | controller | - ' exit 1' 2026-03-20 16:26:25.742360 | controller | - ' fi' 2026-03-20 16:26:25.742364 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.742368 | controller | - ' ' 2026-03-20 16:26:25.742373 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.742377 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.742381 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.742385 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.742389 | controller | --timeout=300s' 2026-03-20 16:26:25.742393 | controller | - ' ' 2026-03-20 16:26:25.742398 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.742402 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.742406 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.742410 | controller | - ' ' 2026-03-20 16:26:25.742414 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.742419 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.742423 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.742427 | controller | - ' ' 2026-03-20 16:26:25.742431 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.742435 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.742439 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.742443 | controller | available"' 2026-03-20 16:26:25.742448 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.742452 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.742456 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.742460 | controller | - ' exit 1' 2026-03-20 16:26:25.742465 | controller | - ' fi' 2026-03-20 16:26:25.742469 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.742473 | controller | - ' ' 2026-03-20 16:26:25.742477 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.742488 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.742493 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.742497 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.742501 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.742508 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.742513 | controller | -- \' 2026-03-20 16:26:25.742517 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.742521 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.742525 | controller | - ' logger.go:42: 16:11:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.742529 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.742534 | controller | - ' logger.go:42: 16:11:52 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.742538 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.742542 | controller | found' 2026-03-20 16:26:25.742546 | controller | - ' logger.go:42: 16:11:53 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.742550 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.742555 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.742559 | controller | - ' ' 2026-03-20 16:26:25.742563 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.742567 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.742571 | controller | - ' ' 2026-03-20 16:26:25.742575 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.742580 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.742584 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.742588 | controller | - ' exit 1' 2026-03-20 16:26:25.742592 | controller | - ' fi' 2026-03-20 16:26:25.742596 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.742601 | controller | - ' ' 2026-03-20 16:26:25.742605 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.742609 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.742613 | controller | | base64 -d)' 2026-03-20 16:26:25.742617 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.742622 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.742626 | controller | - ' exit 1' 2026-03-20 16:26:25.742630 | controller | - ' fi' 2026-03-20 16:26:25.742634 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.742640 | controller | - ' ' 2026-03-20 16:26:25.742645 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.742649 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.742653 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.742657 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.742661 | controller | --timeout=300s' 2026-03-20 16:26:25.742665 | controller | - ' ' 2026-03-20 16:26:25.742670 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.742674 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.742678 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.742682 | controller | - ' ' 2026-03-20 16:26:25.742686 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.742691 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.742709 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.742713 | controller | - ' ' 2026-03-20 16:26:25.742717 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.742722 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.742726 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.742730 | controller | available"' 2026-03-20 16:26:25.742747 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.742753 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.742758 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.742762 | controller | - ' exit 1' 2026-03-20 16:26:25.742766 | controller | - ' fi' 2026-03-20 16:26:25.742770 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.742774 | controller | - ' ' 2026-03-20 16:26:25.742779 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.742790 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.742795 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.742799 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.742804 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.742808 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.742814 | controller | -- \' 2026-03-20 16:26:25.742818 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.742822 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.742826 | controller | - ' logger.go:42: 16:11:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.742831 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.742835 | controller | - ' logger.go:42: 16:11:53 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.742839 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.742843 | controller | found' 2026-03-20 16:26:25.742848 | controller | - ' logger.go:42: 16:11:54 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.742852 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.742856 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.742860 | controller | - ' ' 2026-03-20 16:26:25.742864 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.742869 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.742873 | controller | - ' ' 2026-03-20 16:26:25.742877 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.742881 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.742885 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.742890 | controller | - ' exit 1' 2026-03-20 16:26:25.742894 | controller | - ' fi' 2026-03-20 16:26:25.742898 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.742902 | controller | - ' ' 2026-03-20 16:26:25.742906 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.742911 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.742915 | controller | | base64 -d)' 2026-03-20 16:26:25.742921 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.742925 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.742930 | controller | - ' exit 1' 2026-03-20 16:26:25.742934 | controller | - ' fi' 2026-03-20 16:26:25.742938 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.742942 | controller | - ' ' 2026-03-20 16:26:25.742946 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.742951 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.742955 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.742959 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.742963 | controller | --timeout=300s' 2026-03-20 16:26:25.742969 | controller | - ' ' 2026-03-20 16:26:25.742973 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.742977 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.742989 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.742994 | controller | - ' ' 2026-03-20 16:26:25.742998 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.743002 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.743006 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.743010 | controller | - ' ' 2026-03-20 16:26:25.743014 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.743019 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.743023 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.743027 | controller | available"' 2026-03-20 16:26:25.743031 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.743035 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.743040 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.743044 | controller | - ' exit 1' 2026-03-20 16:26:25.743048 | controller | - ' fi' 2026-03-20 16:26:25.743052 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.743056 | controller | - ' ' 2026-03-20 16:26:25.743061 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.743065 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.743077 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.743081 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.743085 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.743090 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.743094 | controller | -- \' 2026-03-20 16:26:25.743105 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.743109 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.743114 | controller | - ' logger.go:42: 16:11:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.743120 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.743125 | controller | - ' logger.go:42: 16:11:54 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.743129 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.743133 | controller | found' 2026-03-20 16:26:25.743137 | controller | - ' logger.go:42: 16:11:55 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.743141 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.743145 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.743150 | controller | - ' ' 2026-03-20 16:26:25.743154 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.743158 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.743162 | controller | - ' ' 2026-03-20 16:26:25.743167 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.743171 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.743175 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.743232 | controller | - ' exit 1' 2026-03-20 16:26:25.743238 | controller | - ' fi' 2026-03-20 16:26:25.743242 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.743246 | controller | - ' ' 2026-03-20 16:26:25.743250 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.743254 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.743259 | controller | | base64 -d)' 2026-03-20 16:26:25.743263 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.743267 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.743271 | controller | - ' exit 1' 2026-03-20 16:26:25.743275 | controller | - ' fi' 2026-03-20 16:26:25.743280 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.743284 | controller | - ' ' 2026-03-20 16:26:25.743288 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.743292 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.743296 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.743300 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.743305 | controller | --timeout=300s' 2026-03-20 16:26:25.743309 | controller | - ' ' 2026-03-20 16:26:25.743313 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.743317 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.743321 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.743325 | controller | - ' ' 2026-03-20 16:26:25.743330 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.743334 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.743338 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.743342 | controller | - ' ' 2026-03-20 16:26:25.743346 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.743351 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.743355 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.743359 | controller | available"' 2026-03-20 16:26:25.743365 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.743369 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.743374 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.743378 | controller | - ' exit 1' 2026-03-20 16:26:25.743382 | controller | - ' fi' 2026-03-20 16:26:25.743386 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.743390 | controller | - ' ' 2026-03-20 16:26:25.743397 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.743401 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.743413 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.743418 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.743422 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.743426 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.743430 | controller | -- \' 2026-03-20 16:26:25.743435 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.743439 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.743443 | controller | - ' logger.go:42: 16:11:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.743447 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.743452 | controller | - ' logger.go:42: 16:11:55 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.743456 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.743460 | controller | found' 2026-03-20 16:26:25.743464 | controller | - ' logger.go:42: 16:11:56 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.743469 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.743473 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.743477 | controller | - ' ' 2026-03-20 16:26:25.743492 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.743496 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.743500 | controller | - ' ' 2026-03-20 16:26:25.743505 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.743509 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.743513 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.743517 | controller | - ' exit 1' 2026-03-20 16:26:25.743521 | controller | - ' fi' 2026-03-20 16:26:25.743526 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.743530 | controller | - ' ' 2026-03-20 16:26:25.743536 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.743540 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.743544 | controller | | base64 -d)' 2026-03-20 16:26:25.743548 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.743552 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.743557 | controller | - ' exit 1' 2026-03-20 16:26:25.743561 | controller | - ' fi' 2026-03-20 16:26:25.743565 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.743569 | controller | - ' ' 2026-03-20 16:26:25.743573 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.743578 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.743583 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.743588 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.743592 | controller | --timeout=300s' 2026-03-20 16:26:25.743596 | controller | - ' ' 2026-03-20 16:26:25.743600 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.743605 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.743609 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.743613 | controller | - ' ' 2026-03-20 16:26:25.743617 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.743621 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.743626 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.743630 | controller | - ' ' 2026-03-20 16:26:25.743634 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.743638 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.743642 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.743662 | controller | available"' 2026-03-20 16:26:25.743667 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.743671 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.743675 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.743679 | controller | - ' exit 1' 2026-03-20 16:26:25.743684 | controller | - ' fi' 2026-03-20 16:26:25.743688 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.743703 | controller | - ' ' 2026-03-20 16:26:25.743710 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.743714 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.743718 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.743730 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.743735 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.743739 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.743743 | controller | -- \' 2026-03-20 16:26:25.743747 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.743752 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.743756 | controller | - ' logger.go:42: 16:11:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.743760 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.743765 | controller | - ' logger.go:42: 16:11:56 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.743769 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.743773 | controller | found' 2026-03-20 16:26:25.743777 | controller | - ' logger.go:42: 16:11:57 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.743781 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.743786 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.743792 | controller | - ' ' 2026-03-20 16:26:25.743796 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.743800 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.743804 | controller | - ' ' 2026-03-20 16:26:25.743809 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.743813 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.743817 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.743821 | controller | - ' exit 1' 2026-03-20 16:26:25.743825 | controller | - ' fi' 2026-03-20 16:26:25.743830 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.743834 | controller | - ' ' 2026-03-20 16:26:25.743838 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.743842 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.743846 | controller | | base64 -d)' 2026-03-20 16:26:25.743851 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.743855 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.743859 | controller | - ' exit 1' 2026-03-20 16:26:25.743863 | controller | - ' fi' 2026-03-20 16:26:25.743867 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.743872 | controller | - ' ' 2026-03-20 16:26:25.743876 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.743880 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.743884 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.743888 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.743892 | controller | --timeout=300s' 2026-03-20 16:26:25.743897 | controller | - ' ' 2026-03-20 16:26:25.743901 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.743905 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.743909 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.743913 | controller | - ' ' 2026-03-20 16:26:25.743918 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.743922 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.743928 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.743932 | controller | - ' ' 2026-03-20 16:26:25.743936 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.743941 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.743945 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.743949 | controller | available"' 2026-03-20 16:26:25.743953 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.743957 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.743961 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.743966 | controller | - ' exit 1' 2026-03-20 16:26:25.743970 | controller | - ' fi' 2026-03-20 16:26:25.743974 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.743978 | controller | - ' ' 2026-03-20 16:26:25.743982 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.743990 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.743994 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.744007 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.744012 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.744016 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.744020 | controller | -- \' 2026-03-20 16:26:25.744024 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.744028 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.744033 | controller | - ' logger.go:42: 16:11:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.744037 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.744041 | controller | - ' logger.go:42: 16:11:57 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.744046 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.744050 | controller | found' 2026-03-20 16:26:25.744054 | controller | - ' logger.go:42: 16:11:58 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.744058 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.744062 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.744067 | controller | - ' ' 2026-03-20 16:26:25.744071 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.744075 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.744079 | controller | - ' ' 2026-03-20 16:26:25.744083 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.744087 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.744092 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.744098 | controller | - ' exit 1' 2026-03-20 16:26:25.744102 | controller | - ' fi' 2026-03-20 16:26:25.744106 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.744111 | controller | - ' ' 2026-03-20 16:26:25.744115 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.744119 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.744123 | controller | | base64 -d)' 2026-03-20 16:26:25.744128 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.744132 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.744136 | controller | - ' exit 1' 2026-03-20 16:26:25.744140 | controller | - ' fi' 2026-03-20 16:26:25.744144 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.744149 | controller | - ' ' 2026-03-20 16:26:25.744153 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.744157 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.744161 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.744165 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.744169 | controller | --timeout=300s' 2026-03-20 16:26:25.744174 | controller | - ' ' 2026-03-20 16:26:25.744178 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.744182 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.744188 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.744193 | controller | - ' ' 2026-03-20 16:26:25.744197 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.744201 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.744205 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.744209 | controller | - ' ' 2026-03-20 16:26:25.744213 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.744218 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.744222 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.744226 | controller | available"' 2026-03-20 16:26:25.744230 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.744235 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.744239 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.744243 | controller | - ' exit 1' 2026-03-20 16:26:25.744247 | controller | - ' fi' 2026-03-20 16:26:25.744251 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.744255 | controller | - ' ' 2026-03-20 16:26:25.744260 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.744264 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.744268 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.744272 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.744284 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.744288 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.744293 | controller | -- \' 2026-03-20 16:26:25.744297 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.744302 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.744307 | controller | - ' logger.go:42: 16:11:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.744312 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.744316 | controller | - ' logger.go:42: 16:11:59 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.744321 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.744326 | controller | found' 2026-03-20 16:26:25.744331 | controller | - ' logger.go:42: 16:12:00 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.744335 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.744340 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.744344 | controller | - ' ' 2026-03-20 16:26:25.744348 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.744352 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.744356 | controller | - ' ' 2026-03-20 16:26:25.744361 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.744365 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.744369 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.744375 | controller | - ' exit 1' 2026-03-20 16:26:25.744379 | controller | - ' fi' 2026-03-20 16:26:25.744383 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.744388 | controller | - ' ' 2026-03-20 16:26:25.744392 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.744396 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.744400 | controller | | base64 -d)' 2026-03-20 16:26:25.744404 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.744409 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.744413 | controller | - ' exit 1' 2026-03-20 16:26:25.744417 | controller | - ' fi' 2026-03-20 16:26:25.744421 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.744426 | controller | - ' ' 2026-03-20 16:26:25.744430 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.744434 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.744438 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.744443 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.744447 | controller | --timeout=300s' 2026-03-20 16:26:25.744451 | controller | - ' ' 2026-03-20 16:26:25.744455 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.744459 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.744464 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.744470 | controller | - ' ' 2026-03-20 16:26:25.744474 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.744478 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.744482 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.744487 | controller | - ' ' 2026-03-20 16:26:25.744493 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.744497 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.744501 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.744505 | controller | available"' 2026-03-20 16:26:25.744510 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.744514 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.744518 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.744522 | controller | - ' exit 1' 2026-03-20 16:26:25.744526 | controller | - ' fi' 2026-03-20 16:26:25.744531 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.744535 | controller | - ' ' 2026-03-20 16:26:25.744539 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.744543 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.744547 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.744551 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.744564 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.744570 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.744574 | controller | -- \' 2026-03-20 16:26:25.744579 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.744583 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.744587 | controller | - ' logger.go:42: 16:12:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.744591 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.744596 | controller | - ' logger.go:42: 16:12:00 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.744600 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.744604 | controller | found' 2026-03-20 16:26:25.744608 | controller | - ' logger.go:42: 16:12:01 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.744613 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.744617 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.744621 | controller | - ' ' 2026-03-20 16:26:25.744625 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.744629 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.744634 | controller | - ' ' 2026-03-20 16:26:25.744638 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.744642 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.744646 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.744650 | controller | - ' exit 1' 2026-03-20 16:26:25.744655 | controller | - ' fi' 2026-03-20 16:26:25.744659 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.744663 | controller | - ' ' 2026-03-20 16:26:25.744667 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.744671 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.744676 | controller | | base64 -d)' 2026-03-20 16:26:25.744680 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.744684 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.744688 | controller | - ' exit 1' 2026-03-20 16:26:25.744710 | controller | - ' fi' 2026-03-20 16:26:25.744717 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.744721 | controller | - ' ' 2026-03-20 16:26:25.744725 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.744729 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.744734 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.744738 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.744742 | controller | --timeout=300s' 2026-03-20 16:26:25.744746 | controller | - ' ' 2026-03-20 16:26:25.744750 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.744755 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.744759 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.744763 | controller | - ' ' 2026-03-20 16:26:25.744767 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.744771 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.744775 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.744782 | controller | - ' ' 2026-03-20 16:26:25.744786 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.744790 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.744794 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.744799 | controller | available"' 2026-03-20 16:26:25.744803 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.744809 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.744813 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.744817 | controller | - ' exit 1' 2026-03-20 16:26:25.744822 | controller | - ' fi' 2026-03-20 16:26:25.744826 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.744830 | controller | - ' ' 2026-03-20 16:26:25.744834 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.744838 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.744843 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.744847 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.744851 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.744867 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.744873 | controller | -- \' 2026-03-20 16:26:25.744879 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.744885 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.744893 | controller | - ' logger.go:42: 16:12:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.744897 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.744901 | controller | - ' logger.go:42: 16:12:01 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.744906 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.744910 | controller | found' 2026-03-20 16:26:25.744914 | controller | - ' logger.go:42: 16:12:02 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.744918 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.744922 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.744927 | controller | - ' ' 2026-03-20 16:26:25.744931 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.744935 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.744939 | controller | - ' ' 2026-03-20 16:26:25.744944 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.744948 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.744952 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.744956 | controller | - ' exit 1' 2026-03-20 16:26:25.744960 | controller | - ' fi' 2026-03-20 16:26:25.744964 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.744969 | controller | - ' ' 2026-03-20 16:26:25.744973 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.744977 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.744981 | controller | | base64 -d)' 2026-03-20 16:26:25.744985 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.744993 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.744998 | controller | - ' exit 1' 2026-03-20 16:26:25.745004 | controller | - ' fi' 2026-03-20 16:26:25.745010 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.745016 | controller | - ' ' 2026-03-20 16:26:25.745021 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.745027 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.745033 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.745038 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.745042 | controller | --timeout=300s' 2026-03-20 16:26:25.745047 | controller | - ' ' 2026-03-20 16:26:25.745051 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.745055 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.745059 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.745064 | controller | - ' ' 2026-03-20 16:26:25.745068 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.745072 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.745076 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.745080 | controller | - ' ' 2026-03-20 16:26:25.745084 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.745089 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.745093 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.745097 | controller | available"' 2026-03-20 16:26:25.745101 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.745105 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.745110 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.745114 | controller | - ' exit 1' 2026-03-20 16:26:25.745118 | controller | - ' fi' 2026-03-20 16:26:25.745122 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.745126 | controller | - ' ' 2026-03-20 16:26:25.745130 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.745135 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.745139 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.745143 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.745147 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.745161 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.745165 | controller | -- \' 2026-03-20 16:26:25.745169 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.745174 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.745178 | controller | - ' logger.go:42: 16:12:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.745182 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.745191 | controller | - ' logger.go:42: 16:12:02 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.745195 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.745199 | controller | found' 2026-03-20 16:26:25.745203 | controller | - ' logger.go:42: 16:12:03 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.745207 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.745212 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.745219 | controller | - ' ' 2026-03-20 16:26:25.745224 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.745228 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.745232 | controller | - ' ' 2026-03-20 16:26:25.745237 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.745241 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.745245 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.745250 | controller | - ' exit 1' 2026-03-20 16:26:25.745254 | controller | - ' fi' 2026-03-20 16:26:25.745258 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.745262 | controller | - ' ' 2026-03-20 16:26:25.745267 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.745271 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.745275 | controller | | base64 -d)' 2026-03-20 16:26:25.745279 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.745284 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.745288 | controller | - ' exit 1' 2026-03-20 16:26:25.745292 | controller | - ' fi' 2026-03-20 16:26:25.745297 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.745301 | controller | - ' ' 2026-03-20 16:26:25.745305 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.745309 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.745314 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.745318 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.745322 | controller | --timeout=300s' 2026-03-20 16:26:25.745326 | controller | - ' ' 2026-03-20 16:26:25.745330 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.745335 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.745339 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.745343 | controller | - ' ' 2026-03-20 16:26:25.745347 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.745351 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.745356 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.745360 | controller | - ' ' 2026-03-20 16:26:25.745364 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.745368 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.745372 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.745377 | controller | available"' 2026-03-20 16:26:25.745381 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.745387 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.745391 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.745395 | controller | - ' exit 1' 2026-03-20 16:26:25.745400 | controller | - ' fi' 2026-03-20 16:26:25.745404 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.745408 | controller | - ' ' 2026-03-20 16:26:25.745412 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.745416 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.745421 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.745425 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.745429 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.745435 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.745439 | controller | -- \' 2026-03-20 16:26:25.745451 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.745456 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.745460 | controller | - ' logger.go:42: 16:12:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.745464 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.745469 | controller | - ' logger.go:42: 16:12:03 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.745473 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.745477 | controller | found' 2026-03-20 16:26:25.745481 | controller | - ' logger.go:42: 16:12:04 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.745486 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.745490 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.745494 | controller | - ' ' 2026-03-20 16:26:25.745498 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.745502 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.745506 | controller | - ' ' 2026-03-20 16:26:25.745511 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.745515 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.745519 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.745523 | controller | - ' exit 1' 2026-03-20 16:26:25.745527 | controller | - ' fi' 2026-03-20 16:26:25.745532 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.745536 | controller | - ' ' 2026-03-20 16:26:25.745540 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.745544 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.745548 | controller | | base64 -d)' 2026-03-20 16:26:25.745552 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.745556 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.745561 | controller | - ' exit 1' 2026-03-20 16:26:25.745565 | controller | - ' fi' 2026-03-20 16:26:25.745573 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.745579 | controller | - ' ' 2026-03-20 16:26:25.745585 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.745591 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.745599 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.745605 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.745610 | controller | --timeout=300s' 2026-03-20 16:26:25.745614 | controller | - ' ' 2026-03-20 16:26:25.745618 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.745622 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.745627 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.745631 | controller | - ' ' 2026-03-20 16:26:25.745635 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.745639 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.745643 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.745648 | controller | - ' ' 2026-03-20 16:26:25.745652 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.745656 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.745660 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.745664 | controller | available"' 2026-03-20 16:26:25.745669 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.745673 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.745677 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.745681 | controller | - ' exit 1' 2026-03-20 16:26:25.745685 | controller | - ' fi' 2026-03-20 16:26:25.745690 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.745708 | controller | - ' ' 2026-03-20 16:26:25.745714 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.745720 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.745726 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.745731 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.745737 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.745741 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.745745 | controller | -- \' 2026-03-20 16:26:25.745758 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.745763 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.745767 | controller | - ' logger.go:42: 16:12:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.745771 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.745776 | controller | - ' logger.go:42: 16:12:04 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.745783 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.745787 | controller | found' 2026-03-20 16:26:25.745791 | controller | - ' logger.go:42: 16:12:05 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.745795 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.745800 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.745804 | controller | - ' ' 2026-03-20 16:26:25.745810 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.745814 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.745818 | controller | - ' ' 2026-03-20 16:26:25.745823 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.745827 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.745831 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.745835 | controller | - ' exit 1' 2026-03-20 16:26:25.745839 | controller | - ' fi' 2026-03-20 16:26:25.745844 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.745848 | controller | - ' ' 2026-03-20 16:26:25.745852 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.745856 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.745860 | controller | | base64 -d)' 2026-03-20 16:26:25.745865 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.745869 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.745873 | controller | - ' exit 1' 2026-03-20 16:26:25.745877 | controller | - ' fi' 2026-03-20 16:26:25.745882 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.745886 | controller | - ' ' 2026-03-20 16:26:25.745890 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.745894 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.745898 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.745902 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.745906 | controller | --timeout=300s' 2026-03-20 16:26:25.745911 | controller | - ' ' 2026-03-20 16:26:25.745915 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.745919 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.745923 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.745928 | controller | - ' ' 2026-03-20 16:26:25.745932 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.745936 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.745940 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.745944 | controller | - ' ' 2026-03-20 16:26:25.745948 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.745954 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.745958 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.745963 | controller | available"' 2026-03-20 16:26:25.745967 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.745971 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.745975 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.745979 | controller | - ' exit 1' 2026-03-20 16:26:25.745983 | controller | - ' fi' 2026-03-20 16:26:25.745988 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.745992 | controller | - ' ' 2026-03-20 16:26:25.745996 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.746002 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.746006 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.746010 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.746014 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.746019 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.746023 | controller | -- \' 2026-03-20 16:26:25.746027 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.746039 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.746043 | controller | - ' logger.go:42: 16:12:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.746047 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.746051 | controller | - ' logger.go:42: 16:12:05 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.746056 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.746060 | controller | found' 2026-03-20 16:26:25.746064 | controller | - ' logger.go:42: 16:12:06 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.746068 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.746073 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.746077 | controller | - ' ' 2026-03-20 16:26:25.746081 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.746085 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.746089 | controller | - ' ' 2026-03-20 16:26:25.746094 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.746098 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.746102 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.746106 | controller | - ' exit 1' 2026-03-20 16:26:25.746110 | controller | - ' fi' 2026-03-20 16:26:25.746114 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.746119 | controller | - ' ' 2026-03-20 16:26:25.746125 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.746129 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.746133 | controller | | base64 -d)' 2026-03-20 16:26:25.746137 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.746141 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.746146 | controller | - ' exit 1' 2026-03-20 16:26:25.746150 | controller | - ' fi' 2026-03-20 16:26:25.746154 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.746158 | controller | - ' ' 2026-03-20 16:26:25.746163 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.746167 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.746171 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.746175 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.746179 | controller | --timeout=300s' 2026-03-20 16:26:25.746183 | controller | - ' ' 2026-03-20 16:26:25.746188 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.746192 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.746198 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.746202 | controller | - ' ' 2026-03-20 16:26:25.746206 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.746210 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.746214 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.746218 | controller | - ' ' 2026-03-20 16:26:25.746223 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.746227 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.746231 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.746235 | controller | available"' 2026-03-20 16:26:25.746239 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.746244 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.746248 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.746252 | controller | - ' exit 1' 2026-03-20 16:26:25.746256 | controller | - ' fi' 2026-03-20 16:26:25.746260 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.746265 | controller | - ' ' 2026-03-20 16:26:25.746269 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.746273 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.746277 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.746281 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.746286 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.746290 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.746294 | controller | -- \' 2026-03-20 16:26:25.746298 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.746309 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.746313 | controller | - ' logger.go:42: 16:12:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.746318 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.746322 | controller | - ' logger.go:42: 16:12:07 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.746326 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.746330 | controller | found' 2026-03-20 16:26:25.746336 | controller | - ' logger.go:42: 16:12:08 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.746341 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.746345 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.746349 | controller | - ' ' 2026-03-20 16:26:25.746354 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.746358 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.746362 | controller | - ' ' 2026-03-20 16:26:25.746366 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.746370 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.746374 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.746379 | controller | - ' exit 1' 2026-03-20 16:26:25.746384 | controller | - ' fi' 2026-03-20 16:26:25.746389 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.746393 | controller | - ' ' 2026-03-20 16:26:25.746397 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.746401 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.746405 | controller | | base64 -d)' 2026-03-20 16:26:25.746409 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.746414 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.746418 | controller | - ' exit 1' 2026-03-20 16:26:25.746422 | controller | - ' fi' 2026-03-20 16:26:25.746426 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.746430 | controller | - ' ' 2026-03-20 16:26:25.746435 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.746439 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.746443 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.746447 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.746451 | controller | --timeout=300s' 2026-03-20 16:26:25.746456 | controller | - ' ' 2026-03-20 16:26:25.746460 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.746464 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.746468 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.746472 | controller | - ' ' 2026-03-20 16:26:25.746476 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.746481 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.746485 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.746489 | controller | - ' ' 2026-03-20 16:26:25.746493 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.746497 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.746501 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.746505 | controller | available"' 2026-03-20 16:26:25.746510 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.746514 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.746518 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.746522 | controller | - ' exit 1' 2026-03-20 16:26:25.746526 | controller | - ' fi' 2026-03-20 16:26:25.746531 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.746535 | controller | - ' ' 2026-03-20 16:26:25.746539 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.746543 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.746547 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.746552 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.746556 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.746560 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.746566 | controller | -- \' 2026-03-20 16:26:25.746570 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.746574 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.746585 | controller | - ' logger.go:42: 16:12:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.746590 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.746594 | controller | - ' logger.go:42: 16:12:08 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.746598 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.746602 | controller | found' 2026-03-20 16:26:25.746607 | controller | - ' logger.go:42: 16:12:09 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.746611 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.746615 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.746619 | controller | - ' ' 2026-03-20 16:26:25.746623 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.746628 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.746632 | controller | - ' ' 2026-03-20 16:26:25.746636 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.746640 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.746644 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.746649 | controller | - ' exit 1' 2026-03-20 16:26:25.746655 | controller | - ' fi' 2026-03-20 16:26:25.746659 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.746664 | controller | - ' ' 2026-03-20 16:26:25.746668 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.746672 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.746676 | controller | | base64 -d)' 2026-03-20 16:26:25.746680 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.746685 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.746689 | controller | - ' exit 1' 2026-03-20 16:26:25.746704 | controller | - ' fi' 2026-03-20 16:26:25.746710 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.746718 | controller | - ' ' 2026-03-20 16:26:25.746722 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.746726 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.746730 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.746734 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.746739 | controller | --timeout=300s' 2026-03-20 16:26:25.746743 | controller | - ' ' 2026-03-20 16:26:25.746747 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.746751 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.746755 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.746759 | controller | - ' ' 2026-03-20 16:26:25.746764 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.746768 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.746772 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.746778 | controller | - ' ' 2026-03-20 16:26:25.746782 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.746786 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.746790 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.746795 | controller | available"' 2026-03-20 16:26:25.746799 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.746803 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.746807 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.746811 | controller | - ' exit 1' 2026-03-20 16:26:25.746816 | controller | - ' fi' 2026-03-20 16:26:25.746820 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.746824 | controller | - ' ' 2026-03-20 16:26:25.746830 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.746834 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.746838 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.746843 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.746847 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.746851 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.746855 | controller | -- \' 2026-03-20 16:26:25.746859 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.746863 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.746875 | controller | - ' logger.go:42: 16:12:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.746880 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.746884 | controller | - ' logger.go:42: 16:12:09 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.746888 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.746893 | controller | found' 2026-03-20 16:26:25.746897 | controller | - ' logger.go:42: 16:12:10 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.746901 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.746905 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.746910 | controller | - ' ' 2026-03-20 16:26:25.746916 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.746920 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.746924 | controller | - ' ' 2026-03-20 16:26:25.746928 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.746932 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.746937 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.746941 | controller | - ' exit 1' 2026-03-20 16:26:25.746945 | controller | - ' fi' 2026-03-20 16:26:25.746949 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.746953 | controller | - ' ' 2026-03-20 16:26:25.746957 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.746962 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.746966 | controller | | base64 -d)' 2026-03-20 16:26:25.746970 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.746974 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.746980 | controller | - ' exit 1' 2026-03-20 16:26:25.746985 | controller | - ' fi' 2026-03-20 16:26:25.746989 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.746993 | controller | - ' ' 2026-03-20 16:26:25.746997 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.747001 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.747006 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.747010 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.747014 | controller | --timeout=300s' 2026-03-20 16:26:25.747020 | controller | - ' ' 2026-03-20 16:26:25.747024 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.747028 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.747032 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.747037 | controller | - ' ' 2026-03-20 16:26:25.747041 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.747045 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.747049 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.747054 | controller | - ' ' 2026-03-20 16:26:25.747058 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.747062 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.747066 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.747070 | controller | available"' 2026-03-20 16:26:25.747074 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.747079 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.747083 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.747087 | controller | - ' exit 1' 2026-03-20 16:26:25.747093 | controller | - ' fi' 2026-03-20 16:26:25.747097 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.747101 | controller | - ' ' 2026-03-20 16:26:25.747106 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.747110 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.747114 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.747118 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.747122 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.747126 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.747130 | controller | -- \' 2026-03-20 16:26:25.747135 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.747139 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.747143 | controller | - ' logger.go:42: 16:12:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.747154 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.747159 | controller | - ' logger.go:42: 16:12:10 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.747165 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.747169 | controller | found' 2026-03-20 16:26:25.747173 | controller | - ' logger.go:42: 16:12:11 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.747177 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.747182 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.747186 | controller | - ' ' 2026-03-20 16:26:25.747190 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.747194 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.747199 | controller | - ' ' 2026-03-20 16:26:25.747203 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.747207 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.747211 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.747215 | controller | - ' exit 1' 2026-03-20 16:26:25.747220 | controller | - ' fi' 2026-03-20 16:26:25.747224 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.747228 | controller | - ' ' 2026-03-20 16:26:25.747232 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.747236 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.747241 | controller | | base64 -d)' 2026-03-20 16:26:25.747245 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.747249 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.747253 | controller | - ' exit 1' 2026-03-20 16:26:25.747258 | controller | - ' fi' 2026-03-20 16:26:25.747262 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.747266 | controller | - ' ' 2026-03-20 16:26:25.747270 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.747274 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.747278 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.747283 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.747287 | controller | --timeout=300s' 2026-03-20 16:26:25.747291 | controller | - ' ' 2026-03-20 16:26:25.747296 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.747301 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.747307 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.747313 | controller | - ' ' 2026-03-20 16:26:25.747318 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.747324 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.747329 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.747333 | controller | - ' ' 2026-03-20 16:26:25.747337 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.747341 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.747346 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.747350 | controller | available"' 2026-03-20 16:26:25.747354 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.747360 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.747366 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.747371 | controller | - ' exit 1' 2026-03-20 16:26:25.747375 | controller | - ' fi' 2026-03-20 16:26:25.747379 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.747383 | controller | - ' ' 2026-03-20 16:26:25.747387 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.747391 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.747396 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.747400 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.747404 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.747408 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.747412 | controller | -- \' 2026-03-20 16:26:25.747417 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.747421 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.747425 | controller | - ' logger.go:42: 16:12:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.747429 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.747630 | controller | - ' logger.go:42: 16:12:11 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.747634 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.747638 | controller | found' 2026-03-20 16:26:25.747642 | controller | - ' logger.go:42: 16:12:12 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.747647 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.747670 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.747674 | controller | - ' ' 2026-03-20 16:26:25.747679 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.747685 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.747689 | controller | - ' ' 2026-03-20 16:26:25.747693 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.747710 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.747715 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.747719 | controller | - ' exit 1' 2026-03-20 16:26:25.747723 | controller | - ' fi' 2026-03-20 16:26:25.747727 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.747732 | controller | - ' ' 2026-03-20 16:26:25.747736 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.747740 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.747744 | controller | | base64 -d)' 2026-03-20 16:26:25.747749 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.747753 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.747757 | controller | - ' exit 1' 2026-03-20 16:26:25.747761 | controller | - ' fi' 2026-03-20 16:26:25.747765 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.747769 | controller | - ' ' 2026-03-20 16:26:25.747774 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.747778 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.747782 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.747788 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.747793 | controller | --timeout=300s' 2026-03-20 16:26:25.747797 | controller | - ' ' 2026-03-20 16:26:25.747801 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.747805 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.747809 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.747814 | controller | - ' ' 2026-03-20 16:26:25.747818 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.747822 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.747826 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.747830 | controller | - ' ' 2026-03-20 16:26:25.747835 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.747839 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.747843 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.747847 | controller | available"' 2026-03-20 16:26:25.747852 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.747856 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.747860 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.747864 | controller | - ' exit 1' 2026-03-20 16:26:25.747871 | controller | - ' fi' 2026-03-20 16:26:25.747875 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.747879 | controller | - ' ' 2026-03-20 16:26:25.747883 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.747888 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.747892 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.747896 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.747900 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.747904 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.747908 | controller | -- \' 2026-03-20 16:26:25.747913 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.747917 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.747921 | controller | - ' logger.go:42: 16:12:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.747925 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.747929 | controller | - ' logger.go:42: 16:12:12 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.747942 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.747946 | controller | found' 2026-03-20 16:26:25.747950 | controller | - ' logger.go:42: 16:12:13 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.747954 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.747959 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.747963 | controller | - ' ' 2026-03-20 16:26:25.747967 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.747973 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.747977 | controller | - ' ' 2026-03-20 16:26:25.747982 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.747986 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.747990 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.747994 | controller | - ' exit 1' 2026-03-20 16:26:25.747999 | controller | - ' fi' 2026-03-20 16:26:25.748003 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.748007 | controller | - ' ' 2026-03-20 16:26:25.748011 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.748016 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.748020 | controller | | base64 -d)' 2026-03-20 16:26:25.748024 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.748028 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.748032 | controller | - ' exit 1' 2026-03-20 16:26:25.748037 | controller | - ' fi' 2026-03-20 16:26:25.748041 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.748045 | controller | - ' ' 2026-03-20 16:26:25.748049 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.748053 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.748058 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.748062 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.748066 | controller | --timeout=300s' 2026-03-20 16:26:25.748070 | controller | - ' ' 2026-03-20 16:26:25.748074 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.748078 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.748083 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.748087 | controller | - ' ' 2026-03-20 16:26:25.748091 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.748095 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.748099 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.748104 | controller | - ' ' 2026-03-20 16:26:25.748108 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.748112 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.748116 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.748121 | controller | available"' 2026-03-20 16:26:25.748125 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.748129 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.748133 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.748137 | controller | - ' exit 1' 2026-03-20 16:26:25.748141 | controller | - ' fi' 2026-03-20 16:26:25.748146 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.748150 | controller | - ' ' 2026-03-20 16:26:25.748154 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.748158 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.748164 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.748169 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.748176 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.748180 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.748185 | controller | -- \' 2026-03-20 16:26:25.748189 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.748193 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.748197 | controller | - ' logger.go:42: 16:12:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.748202 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.748206 | controller | - ' logger.go:42: 16:12:13 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.748217 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.748221 | controller | found' 2026-03-20 16:26:25.748225 | controller | - ' logger.go:42: 16:12:14 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.748230 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.748234 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.748238 | controller | - ' ' 2026-03-20 16:26:25.748242 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.748246 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.748251 | controller | - ' ' 2026-03-20 16:26:25.748257 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.748261 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.748265 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.748271 | controller | - ' exit 1' 2026-03-20 16:26:25.748275 | controller | - ' fi' 2026-03-20 16:26:25.748279 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.748284 | controller | - ' ' 2026-03-20 16:26:25.748288 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.748292 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.748296 | controller | | base64 -d)' 2026-03-20 16:26:25.748300 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.748305 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.748309 | controller | - ' exit 1' 2026-03-20 16:26:25.748313 | controller | - ' fi' 2026-03-20 16:26:25.748317 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.748321 | controller | - ' ' 2026-03-20 16:26:25.748326 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.748330 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.748334 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.748338 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.748342 | controller | --timeout=300s' 2026-03-20 16:26:25.748346 | controller | - ' ' 2026-03-20 16:26:25.748351 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.748355 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.748359 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.748365 | controller | - ' ' 2026-03-20 16:26:25.748369 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.748374 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.748378 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.748382 | controller | - ' ' 2026-03-20 16:26:25.748386 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.748390 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.748394 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.748399 | controller | available"' 2026-03-20 16:26:25.748403 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.748407 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.748411 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.748415 | controller | - ' exit 1' 2026-03-20 16:26:25.748420 | controller | - ' fi' 2026-03-20 16:26:25.748424 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.748428 | controller | - ' ' 2026-03-20 16:26:25.748432 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.748437 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.748441 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.748445 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.748449 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.748453 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.748458 | controller | -- \' 2026-03-20 16:26:25.748462 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.748466 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.748470 | controller | - ' logger.go:42: 16:12:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.748474 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.748478 | controller | - ' logger.go:42: 16:12:15 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.748483 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.748487 | controller | found' 2026-03-20 16:26:25.748499 | controller | - ' logger.go:42: 16:12:16 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.748503 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.748507 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.748512 | controller | - ' ' 2026-03-20 16:26:25.748516 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.748520 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.748524 | controller | - ' ' 2026-03-20 16:26:25.748529 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.748533 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.748537 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.748541 | controller | - ' exit 1' 2026-03-20 16:26:25.748546 | controller | - ' fi' 2026-03-20 16:26:25.748552 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.748556 | controller | - ' ' 2026-03-20 16:26:25.748560 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.748564 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.748568 | controller | | base64 -d)' 2026-03-20 16:26:25.748573 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.748577 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.748581 | controller | - ' exit 1' 2026-03-20 16:26:25.748585 | controller | - ' fi' 2026-03-20 16:26:25.748589 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.748594 | controller | - ' ' 2026-03-20 16:26:25.748598 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.748602 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.748606 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.748610 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.748615 | controller | --timeout=300s' 2026-03-20 16:26:25.748619 | controller | - ' ' 2026-03-20 16:26:25.748623 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.748627 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.748631 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.748636 | controller | - ' ' 2026-03-20 16:26:25.748640 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.748644 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.748648 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.748652 | controller | - ' ' 2026-03-20 16:26:25.748656 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.748661 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.748665 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.748669 | controller | available"' 2026-03-20 16:26:25.748673 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.748677 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.748682 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.748686 | controller | - ' exit 1' 2026-03-20 16:26:25.748690 | controller | - ' fi' 2026-03-20 16:26:25.748707 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.748712 | controller | - ' ' 2026-03-20 16:26:25.748716 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.748720 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.748724 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.748728 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.748733 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.748737 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.748743 | controller | -- \' 2026-03-20 16:26:25.748747 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.748752 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.748756 | controller | - ' logger.go:42: 16:12:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.748760 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.748764 | controller | - ' logger.go:42: 16:12:16 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.748768 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.748773 | controller | found' 2026-03-20 16:26:25.748784 | controller | - ' logger.go:42: 16:12:17 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.748788 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.748793 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.748797 | controller | - ' ' 2026-03-20 16:26:25.748801 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.748805 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.748809 | controller | - ' ' 2026-03-20 16:26:25.748814 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.748818 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.748822 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.748827 | controller | - ' exit 1' 2026-03-20 16:26:25.748831 | controller | - ' fi' 2026-03-20 16:26:25.748835 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.748841 | controller | - ' ' 2026-03-20 16:26:25.748846 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.748850 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.748854 | controller | | base64 -d)' 2026-03-20 16:26:25.748858 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.748864 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.748868 | controller | - ' exit 1' 2026-03-20 16:26:25.748872 | controller | - ' fi' 2026-03-20 16:26:25.748877 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.748881 | controller | - ' ' 2026-03-20 16:26:25.748885 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.748889 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.748893 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.748898 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.748902 | controller | --timeout=300s' 2026-03-20 16:26:25.748908 | controller | - ' ' 2026-03-20 16:26:25.748912 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.748916 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.748921 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.748925 | controller | - ' ' 2026-03-20 16:26:25.748929 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.748933 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.748937 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.748941 | controller | - ' ' 2026-03-20 16:26:25.748947 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.748952 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.748956 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.748960 | controller | available"' 2026-03-20 16:26:25.748964 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.748968 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.748973 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.748977 | controller | - ' exit 1' 2026-03-20 16:26:25.748981 | controller | - ' fi' 2026-03-20 16:26:25.748985 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.748989 | controller | - ' ' 2026-03-20 16:26:25.748993 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.748998 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.749002 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.749006 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.749010 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.749014 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.749018 | controller | -- \' 2026-03-20 16:26:25.749023 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.749027 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.749031 | controller | - ' logger.go:42: 16:12:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.749035 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.749041 | controller | - ' logger.go:42: 16:12:17 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.749046 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.749050 | controller | found' 2026-03-20 16:26:25.749054 | controller | - ' logger.go:42: 16:12:18 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.749066 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.749071 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.749075 | controller | - ' ' 2026-03-20 16:26:25.749079 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.749083 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.749087 | controller | - ' ' 2026-03-20 16:26:25.749092 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.749096 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.749100 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.749105 | controller | - ' exit 1' 2026-03-20 16:26:25.749109 | controller | - ' fi' 2026-03-20 16:26:25.749113 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.749117 | controller | - ' ' 2026-03-20 16:26:25.749121 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.749126 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.749130 | controller | | base64 -d)' 2026-03-20 16:26:25.749134 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.749138 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.749144 | controller | - ' exit 1' 2026-03-20 16:26:25.749148 | controller | - ' fi' 2026-03-20 16:26:25.749153 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.749157 | controller | - ' ' 2026-03-20 16:26:25.749161 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.749165 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.749170 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.749174 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.749178 | controller | --timeout=300s' 2026-03-20 16:26:25.749182 | controller | - ' ' 2026-03-20 16:26:25.749186 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.749190 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.749195 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.749199 | controller | - ' ' 2026-03-20 16:26:25.749203 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.749209 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.749213 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.749218 | controller | - ' ' 2026-03-20 16:26:25.749222 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.749226 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.749230 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.749234 | controller | available"' 2026-03-20 16:26:25.749239 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.749243 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.749247 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.749251 | controller | - ' exit 1' 2026-03-20 16:26:25.749256 | controller | - ' fi' 2026-03-20 16:26:25.749260 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.749264 | controller | - ' ' 2026-03-20 16:26:25.749268 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.749272 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.749276 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.749281 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.749285 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.749289 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.749293 | controller | -- \' 2026-03-20 16:26:25.749299 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.749303 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.749308 | controller | - ' logger.go:42: 16:12:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.749312 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.749316 | controller | - ' logger.go:42: 16:12:18 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.749322 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.749326 | controller | found' 2026-03-20 16:26:25.749330 | controller | - ' logger.go:42: 16:12:19 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.749335 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.749339 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.749350 | controller | - ' ' 2026-03-20 16:26:25.749355 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.749359 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.749363 | controller | - ' ' 2026-03-20 16:26:25.749367 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.749371 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.749376 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.749380 | controller | - ' exit 1' 2026-03-20 16:26:25.749384 | controller | - ' fi' 2026-03-20 16:26:25.749388 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.749393 | controller | - ' ' 2026-03-20 16:26:25.749397 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.749401 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.749405 | controller | | base64 -d)' 2026-03-20 16:26:25.749409 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.749413 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.749418 | controller | - ' exit 1' 2026-03-20 16:26:25.749422 | controller | - ' fi' 2026-03-20 16:26:25.749426 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.749430 | controller | - ' ' 2026-03-20 16:26:25.749434 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.749439 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.749445 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.749449 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.749453 | controller | --timeout=300s' 2026-03-20 16:26:25.749457 | controller | - ' ' 2026-03-20 16:26:25.749461 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.749466 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.749470 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.749474 | controller | - ' ' 2026-03-20 16:26:25.749478 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.749482 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.749487 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.749491 | controller | - ' ' 2026-03-20 16:26:25.749495 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.749499 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.749506 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.749511 | controller | available"' 2026-03-20 16:26:25.749515 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.749519 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.749525 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.749529 | controller | - ' exit 1' 2026-03-20 16:26:25.749534 | controller | - ' fi' 2026-03-20 16:26:25.749538 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.749542 | controller | - ' ' 2026-03-20 16:26:25.749546 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.749550 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.749555 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.749559 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.749563 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.749567 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.749571 | controller | -- \' 2026-03-20 16:26:25.749576 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.749580 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.749584 | controller | - ' logger.go:42: 16:12:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.749588 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.749592 | controller | - ' logger.go:42: 16:12:19 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.749597 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.749601 | controller | found' 2026-03-20 16:26:25.749605 | controller | - ' logger.go:42: 16:12:20 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.749609 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.749613 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.749617 | controller | - ' ' 2026-03-20 16:26:25.749628 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.749633 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.749637 | controller | - ' ' 2026-03-20 16:26:25.749641 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.749645 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.749650 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.749654 | controller | - ' exit 1' 2026-03-20 16:26:25.749658 | controller | - ' fi' 2026-03-20 16:26:25.749663 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.749667 | controller | - ' ' 2026-03-20 16:26:25.749671 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.749675 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.749679 | controller | | base64 -d)' 2026-03-20 16:26:25.749684 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.749688 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.749704 | controller | - ' exit 1' 2026-03-20 16:26:25.749710 | controller | - ' fi' 2026-03-20 16:26:25.749715 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.749719 | controller | - ' ' 2026-03-20 16:26:25.749723 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.749727 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.749731 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.749738 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.749742 | controller | --timeout=300s' 2026-03-20 16:26:25.749746 | controller | - ' ' 2026-03-20 16:26:25.749750 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.749754 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.749759 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.749763 | controller | - ' ' 2026-03-20 16:26:25.749767 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.749771 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.749775 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.749780 | controller | - ' ' 2026-03-20 16:26:25.749784 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.749788 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.749792 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.749797 | controller | available"' 2026-03-20 16:26:25.749801 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.749805 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.749809 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.749813 | controller | - ' exit 1' 2026-03-20 16:26:25.749818 | controller | - ' fi' 2026-03-20 16:26:25.749822 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.749826 | controller | - ' ' 2026-03-20 16:26:25.749830 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.749834 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.749839 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.749843 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.749847 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.749851 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.749855 | controller | -- \' 2026-03-20 16:26:25.749860 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.749864 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.749868 | controller | - ' logger.go:42: 16:12:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.749872 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.749877 | controller | - ' logger.go:42: 16:12:20 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.749881 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.749885 | controller | found' 2026-03-20 16:26:25.749889 | controller | - ' logger.go:42: 16:12:21 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.749893 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.749898 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.749902 | controller | - ' ' 2026-03-20 16:26:25.749906 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.749918 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.749924 | controller | - ' ' 2026-03-20 16:26:25.749928 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.749932 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.749937 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.749941 | controller | - ' exit 1' 2026-03-20 16:26:25.749945 | controller | - ' fi' 2026-03-20 16:26:25.749949 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.749954 | controller | - ' ' 2026-03-20 16:26:25.749960 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.749965 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.749969 | controller | | base64 -d)' 2026-03-20 16:26:25.749973 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.749977 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.749982 | controller | - ' exit 1' 2026-03-20 16:26:25.749986 | controller | - ' fi' 2026-03-20 16:26:25.749990 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.749994 | controller | - ' ' 2026-03-20 16:26:25.749999 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.750003 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.750007 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.750013 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.750017 | controller | --timeout=300s' 2026-03-20 16:26:25.750021 | controller | - ' ' 2026-03-20 16:26:25.750025 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.750029 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.750034 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.750038 | controller | - ' ' 2026-03-20 16:26:25.750042 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.750046 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.750050 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.750055 | controller | - ' ' 2026-03-20 16:26:25.750059 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.750063 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.750067 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.750072 | controller | available"' 2026-03-20 16:26:25.750076 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.750082 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.750086 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.750090 | controller | - ' exit 1' 2026-03-20 16:26:25.750094 | controller | - ' fi' 2026-03-20 16:26:25.750099 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.750103 | controller | - ' ' 2026-03-20 16:26:25.750107 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.750111 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.750115 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.750131 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.750135 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.750139 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.750145 | controller | -- \' 2026-03-20 16:26:25.750150 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.750154 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.750158 | controller | - ' logger.go:42: 16:12:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.750162 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.750166 | controller | - ' logger.go:42: 16:12:22 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.750171 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.750175 | controller | found' 2026-03-20 16:26:25.750179 | controller | - ' logger.go:42: 16:12:23 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.750183 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.750187 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.750192 | controller | - ' ' 2026-03-20 16:26:25.750196 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.750207 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.750212 | controller | - ' ' 2026-03-20 16:26:25.750216 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.750220 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.750225 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.750229 | controller | - ' exit 1' 2026-03-20 16:26:25.750233 | controller | - ' fi' 2026-03-20 16:26:25.750237 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.750241 | controller | - ' ' 2026-03-20 16:26:25.750246 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.750250 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.750254 | controller | | base64 -d)' 2026-03-20 16:26:25.750258 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.750262 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.750267 | controller | - ' exit 1' 2026-03-20 16:26:25.750271 | controller | - ' fi' 2026-03-20 16:26:25.750275 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.750279 | controller | - ' ' 2026-03-20 16:26:25.750284 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.750288 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.750292 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.750296 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.750300 | controller | --timeout=300s' 2026-03-20 16:26:25.750304 | controller | - ' ' 2026-03-20 16:26:25.750309 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.750313 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.750317 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.750323 | controller | - ' ' 2026-03-20 16:26:25.750327 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.750331 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.750335 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.750340 | controller | - ' ' 2026-03-20 16:26:25.750344 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.750348 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.750352 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.750356 | controller | available"' 2026-03-20 16:26:25.750361 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.750365 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.750369 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.750373 | controller | - ' exit 1' 2026-03-20 16:26:25.750377 | controller | - ' fi' 2026-03-20 16:26:25.750382 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.750386 | controller | - ' ' 2026-03-20 16:26:25.750390 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.750394 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.750398 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.750403 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.750407 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.750411 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.750415 | controller | -- \' 2026-03-20 16:26:25.750419 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.750423 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.750428 | controller | - ' logger.go:42: 16:12:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.750432 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.750436 | controller | - ' logger.go:42: 16:12:23 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.750440 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.750444 | controller | found' 2026-03-20 16:26:25.750449 | controller | - ' logger.go:42: 16:12:24 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.750453 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.750457 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.750461 | controller | - ' ' 2026-03-20 16:26:25.750465 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.750469 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.750474 | controller | - ' ' 2026-03-20 16:26:25.750485 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.750489 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.750493 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.750498 | controller | - ' exit 1' 2026-03-20 16:26:25.750502 | controller | - ' fi' 2026-03-20 16:26:25.750506 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.750512 | controller | - ' ' 2026-03-20 16:26:25.750516 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.750521 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.750525 | controller | | base64 -d)' 2026-03-20 16:26:25.750529 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.750533 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.750537 | controller | - ' exit 1' 2026-03-20 16:26:25.750542 | controller | - ' fi' 2026-03-20 16:26:25.750546 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.750550 | controller | - ' ' 2026-03-20 16:26:25.750554 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.750558 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.750563 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.750567 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.750571 | controller | --timeout=300s' 2026-03-20 16:26:25.750575 | controller | - ' ' 2026-03-20 16:26:25.750579 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.750584 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.750588 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.750592 | controller | - ' ' 2026-03-20 16:26:25.750596 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.750601 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.750605 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.750611 | controller | - ' ' 2026-03-20 16:26:25.750617 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.750621 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.750625 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.750630 | controller | available"' 2026-03-20 16:26:25.750634 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.750638 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.750642 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.750646 | controller | - ' exit 1' 2026-03-20 16:26:25.750650 | controller | - ' fi' 2026-03-20 16:26:25.750655 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.750659 | controller | - ' ' 2026-03-20 16:26:25.750663 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.750667 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.750672 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.750676 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.750680 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.750684 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.750688 | controller | -- \' 2026-03-20 16:26:25.750703 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.750712 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.750716 | controller | - ' logger.go:42: 16:12:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.750720 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.750724 | controller | - ' logger.go:42: 16:12:24 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.750728 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.750733 | controller | found' 2026-03-20 16:26:25.750737 | controller | - ' logger.go:42: 16:12:25 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.750741 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.750745 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.750750 | controller | - ' ' 2026-03-20 16:26:25.750754 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.750758 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.750762 | controller | - ' ' 2026-03-20 16:26:25.750774 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.750779 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.750783 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.750787 | controller | - ' exit 1' 2026-03-20 16:26:25.750792 | controller | - ' fi' 2026-03-20 16:26:25.750796 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.750800 | controller | - ' ' 2026-03-20 16:26:25.750804 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.750809 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.750813 | controller | | base64 -d)' 2026-03-20 16:26:25.750817 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.750821 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.750825 | controller | - ' exit 1' 2026-03-20 16:26:25.750830 | controller | - ' fi' 2026-03-20 16:26:25.750834 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.750840 | controller | - ' ' 2026-03-20 16:26:25.750844 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.750849 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.750853 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.750857 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.750861 | controller | --timeout=300s' 2026-03-20 16:26:25.750865 | controller | - ' ' 2026-03-20 16:26:25.750870 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.750874 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.750878 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.750882 | controller | - ' ' 2026-03-20 16:26:25.750887 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.750891 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.750895 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.750899 | controller | - ' ' 2026-03-20 16:26:25.750903 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.750910 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.750915 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.750919 | controller | available"' 2026-03-20 16:26:25.750923 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.750927 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.750932 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.750936 | controller | - ' exit 1' 2026-03-20 16:26:25.750940 | controller | - ' fi' 2026-03-20 16:26:25.750944 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.750949 | controller | - ' ' 2026-03-20 16:26:25.750955 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.750959 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.750963 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.750967 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.750971 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.750976 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.750980 | controller | -- \' 2026-03-20 16:26:25.750984 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.750988 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.750992 | controller | - ' logger.go:42: 16:12:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.750997 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.751001 | controller | - ' logger.go:42: 16:12:25 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.751005 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.751009 | controller | found' 2026-03-20 16:26:25.751013 | controller | - ' logger.go:42: 16:12:26 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.751018 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.751022 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.751026 | controller | - ' ' 2026-03-20 16:26:25.751030 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.751035 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.751039 | controller | - ' ' 2026-03-20 16:26:25.751043 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.751054 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.751059 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.751063 | controller | - ' exit 1' 2026-03-20 16:26:25.751067 | controller | - ' fi' 2026-03-20 16:26:25.751071 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.751076 | controller | - ' ' 2026-03-20 16:26:25.751080 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.751084 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.751088 | controller | | base64 -d)' 2026-03-20 16:26:25.751092 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.751097 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.751101 | controller | - ' exit 1' 2026-03-20 16:26:25.751107 | controller | - ' fi' 2026-03-20 16:26:25.751111 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.751115 | controller | - ' ' 2026-03-20 16:26:25.751119 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.751124 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.751128 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.751132 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.751136 | controller | --timeout=300s' 2026-03-20 16:26:25.751142 | controller | - ' ' 2026-03-20 16:26:25.751146 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.751150 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.751155 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.751159 | controller | - ' ' 2026-03-20 16:26:25.751163 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.751167 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.751171 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.751176 | controller | - ' ' 2026-03-20 16:26:25.751180 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.751184 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.751190 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.751194 | controller | available"' 2026-03-20 16:26:25.751199 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.751203 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.751207 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.751211 | controller | - ' exit 1' 2026-03-20 16:26:25.751223 | controller | - ' fi' 2026-03-20 16:26:25.751228 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.751232 | controller | - ' ' 2026-03-20 16:26:25.751236 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.751240 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.751245 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.751249 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.751253 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.751257 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.751262 | controller | -- \' 2026-03-20 16:26:25.751266 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.751270 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.751274 | controller | - ' logger.go:42: 16:12:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.751278 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.751282 | controller | - ' logger.go:42: 16:12:26 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.751287 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.751293 | controller | found' 2026-03-20 16:26:25.751297 | controller | - ' logger.go:42: 16:12:27 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.751301 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.751306 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.751310 | controller | - ' ' 2026-03-20 16:26:25.751314 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.751318 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.751322 | controller | - ' ' 2026-03-20 16:26:25.751327 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.751331 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.751342 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.751347 | controller | - ' exit 1' 2026-03-20 16:26:25.751351 | controller | - ' fi' 2026-03-20 16:26:25.751355 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.751360 | controller | - ' ' 2026-03-20 16:26:25.751366 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.751370 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.751374 | controller | | base64 -d)' 2026-03-20 16:26:25.751378 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.751382 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.751387 | controller | - ' exit 1' 2026-03-20 16:26:25.751391 | controller | - ' fi' 2026-03-20 16:26:25.751395 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.751399 | controller | - ' ' 2026-03-20 16:26:25.751404 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.751408 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.751412 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.751416 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.751420 | controller | --timeout=300s' 2026-03-20 16:26:25.751425 | controller | - ' ' 2026-03-20 16:26:25.751429 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.751433 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.751437 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.751441 | controller | - ' ' 2026-03-20 16:26:25.751446 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.751450 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.751454 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.751458 | controller | - ' ' 2026-03-20 16:26:25.751462 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.751466 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.751471 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.751475 | controller | available"' 2026-03-20 16:26:25.751479 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.751483 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.751487 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.751565 | controller | - ' exit 1' 2026-03-20 16:26:25.751570 | controller | - ' fi' 2026-03-20 16:26:25.751574 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.751578 | controller | - ' ' 2026-03-20 16:26:25.751582 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.751586 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.751590 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.751595 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.751599 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.751603 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.751607 | controller | -- \' 2026-03-20 16:26:25.751611 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.751616 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.751620 | controller | - ' logger.go:42: 16:12:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.751624 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.751628 | controller | - ' logger.go:42: 16:12:27 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.751632 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.751637 | controller | found' 2026-03-20 16:26:25.751641 | controller | - ' logger.go:42: 16:12:28 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.751645 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.751649 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.751653 | controller | - ' ' 2026-03-20 16:26:25.751676 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.751681 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.751685 | controller | - ' ' 2026-03-20 16:26:25.751689 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.751703 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.751709 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.751721 | controller | - ' exit 1' 2026-03-20 16:26:25.751726 | controller | - ' fi' 2026-03-20 16:26:25.751730 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.751734 | controller | - ' ' 2026-03-20 16:26:25.751738 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.751742 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.751747 | controller | | base64 -d)' 2026-03-20 16:26:25.751751 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.751757 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.751762 | controller | - ' exit 1' 2026-03-20 16:26:25.751766 | controller | - ' fi' 2026-03-20 16:26:25.751770 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.751774 | controller | - ' ' 2026-03-20 16:26:25.751779 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.751783 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.751787 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.751791 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.751797 | controller | --timeout=300s' 2026-03-20 16:26:25.751801 | controller | - ' ' 2026-03-20 16:26:25.751805 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.751810 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.751814 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.751818 | controller | - ' ' 2026-03-20 16:26:25.751822 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.751826 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.751831 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.751835 | controller | - ' ' 2026-03-20 16:26:25.751839 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.751843 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.751847 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.751852 | controller | available"' 2026-03-20 16:26:25.751856 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.751862 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.751866 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.751870 | controller | - ' exit 1' 2026-03-20 16:26:25.751874 | controller | - ' fi' 2026-03-20 16:26:25.751878 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.751883 | controller | - ' ' 2026-03-20 16:26:25.751887 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.751891 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.751895 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.751899 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.751904 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.751908 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.751912 | controller | -- \' 2026-03-20 16:26:25.751916 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.751920 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.751924 | controller | - ' logger.go:42: 16:12:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.751929 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.751933 | controller | - ' logger.go:42: 16:12:28 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.751937 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.751941 | controller | found' 2026-03-20 16:26:25.751945 | controller | - ' logger.go:42: 16:12:29 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.751950 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.751954 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.751958 | controller | - ' ' 2026-03-20 16:26:25.751962 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.751966 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.751972 | controller | - ' ' 2026-03-20 16:26:25.751976 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.751981 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.751985 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.751989 | controller | - ' exit 1' 2026-03-20 16:26:25.751993 | controller | - ' fi' 2026-03-20 16:26:25.752007 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.752012 | controller | - ' ' 2026-03-20 16:26:25.752016 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.752021 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.752025 | controller | | base64 -d)' 2026-03-20 16:26:25.752029 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.752033 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.752037 | controller | - ' exit 1' 2026-03-20 16:26:25.752042 | controller | - ' fi' 2026-03-20 16:26:25.752046 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.752050 | controller | - ' ' 2026-03-20 16:26:25.752054 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.752058 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.752063 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.752067 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.752071 | controller | --timeout=300s' 2026-03-20 16:26:25.752075 | controller | - ' ' 2026-03-20 16:26:25.752079 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.752084 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.752088 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.752092 | controller | - ' ' 2026-03-20 16:26:25.752098 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.752103 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.752107 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.752111 | controller | - ' ' 2026-03-20 16:26:25.752115 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.752120 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.752124 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.752128 | controller | available"' 2026-03-20 16:26:25.752132 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.752136 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.752141 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.752145 | controller | - ' exit 1' 2026-03-20 16:26:25.752149 | controller | - ' fi' 2026-03-20 16:26:25.752153 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.752157 | controller | - ' ' 2026-03-20 16:26:25.752162 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.752166 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.752170 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.752176 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.752180 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.752185 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.752189 | controller | -- \' 2026-03-20 16:26:25.752193 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.752197 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.752205 | controller | - ' logger.go:42: 16:12:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.752209 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.752213 | controller | - ' logger.go:42: 16:12:30 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.752217 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.752222 | controller | found' 2026-03-20 16:26:25.752226 | controller | - ' logger.go:42: 16:12:31 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.752230 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.752234 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.752238 | controller | - ' ' 2026-03-20 16:26:25.752243 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.752247 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.752251 | controller | - ' ' 2026-03-20 16:26:25.752255 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.752259 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.752264 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.752268 | controller | - ' exit 1' 2026-03-20 16:26:25.752272 | controller | - ' fi' 2026-03-20 16:26:25.752276 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.752280 | controller | - ' ' 2026-03-20 16:26:25.752292 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.752296 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.752301 | controller | | base64 -d)' 2026-03-20 16:26:25.752305 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.752309 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.752315 | controller | - ' exit 1' 2026-03-20 16:26:25.752319 | controller | - ' fi' 2026-03-20 16:26:25.752324 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.752328 | controller | - ' ' 2026-03-20 16:26:25.752332 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.752336 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.752341 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.752345 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.752349 | controller | --timeout=300s' 2026-03-20 16:26:25.752353 | controller | - ' ' 2026-03-20 16:26:25.752357 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.752362 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.752366 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.752370 | controller | - ' ' 2026-03-20 16:26:25.752376 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.752380 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.752384 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.752388 | controller | - ' ' 2026-03-20 16:26:25.752393 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.752397 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.752401 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.752405 | controller | available"' 2026-03-20 16:26:25.752409 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.752414 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.752418 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.752422 | controller | - ' exit 1' 2026-03-20 16:26:25.752426 | controller | - ' fi' 2026-03-20 16:26:25.752430 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.752434 | controller | - ' ' 2026-03-20 16:26:25.752440 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.752444 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.752449 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.752453 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.752457 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.752461 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.752466 | controller | -- \' 2026-03-20 16:26:25.752470 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.752474 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.752478 | controller | - ' logger.go:42: 16:12:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.752482 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.752487 | controller | - ' logger.go:42: 16:12:31 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.752491 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.752495 | controller | found' 2026-03-20 16:26:25.752499 | controller | - ' logger.go:42: 16:12:32 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.752503 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.752508 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.752514 | controller | - ' ' 2026-03-20 16:26:25.752518 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.752523 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.752527 | controller | - ' ' 2026-03-20 16:26:25.752531 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.752535 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.752540 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.752544 | controller | - ' exit 1' 2026-03-20 16:26:25.752548 | controller | - ' fi' 2026-03-20 16:26:25.752553 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.752557 | controller | - ' ' 2026-03-20 16:26:25.752571 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.752576 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.752580 | controller | | base64 -d)' 2026-03-20 16:26:25.752584 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.752588 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.752593 | controller | - ' exit 1' 2026-03-20 16:26:25.752597 | controller | - ' fi' 2026-03-20 16:26:25.752601 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.752605 | controller | - ' ' 2026-03-20 16:26:25.752610 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.752614 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.752618 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.752622 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.752626 | controller | --timeout=300s' 2026-03-20 16:26:25.752631 | controller | - ' ' 2026-03-20 16:26:25.752635 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.752639 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.752643 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.752648 | controller | - ' ' 2026-03-20 16:26:25.752652 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.752656 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.752660 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.752665 | controller | - ' ' 2026-03-20 16:26:25.752669 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.752673 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.752677 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.752681 | controller | available"' 2026-03-20 16:26:25.752686 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.752703 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.752710 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.752715 | controller | - ' exit 1' 2026-03-20 16:26:25.752719 | controller | - ' fi' 2026-03-20 16:26:25.752723 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.752728 | controller | - ' ' 2026-03-20 16:26:25.752732 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.752736 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.752741 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.752745 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.752749 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.752753 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.752758 | controller | -- \' 2026-03-20 16:26:25.752762 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.752768 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.752773 | controller | - ' logger.go:42: 16:12:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.752777 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.752782 | controller | - ' logger.go:42: 16:12:32 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.752786 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.752790 | controller | found' 2026-03-20 16:26:25.752794 | controller | - ' logger.go:42: 16:12:33 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.752799 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.752803 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.752807 | controller | - ' ' 2026-03-20 16:26:25.752812 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.752816 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.752820 | controller | - ' ' 2026-03-20 16:26:25.752824 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.752829 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.752833 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.752837 | controller | - ' exit 1' 2026-03-20 16:26:25.752842 | controller | - ' fi' 2026-03-20 16:26:25.752846 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.752850 | controller | - ' ' 2026-03-20 16:26:25.752854 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.752870 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.752874 | controller | | base64 -d)' 2026-03-20 16:26:25.752879 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.752883 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.752887 | controller | - ' exit 1' 2026-03-20 16:26:25.752892 | controller | - ' fi' 2026-03-20 16:26:25.752896 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.752900 | controller | - ' ' 2026-03-20 16:26:25.752905 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.752909 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.752913 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.752918 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.752922 | controller | --timeout=300s' 2026-03-20 16:26:25.752926 | controller | - ' ' 2026-03-20 16:26:25.752930 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.752934 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.752939 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.752943 | controller | - ' ' 2026-03-20 16:26:25.752947 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.752951 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.752956 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.752960 | controller | - ' ' 2026-03-20 16:26:25.752964 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.752968 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.752974 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.752979 | controller | available"' 2026-03-20 16:26:25.752983 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.752987 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.752991 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.752996 | controller | - ' exit 1' 2026-03-20 16:26:25.753000 | controller | - ' fi' 2026-03-20 16:26:25.753004 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.753009 | controller | - ' ' 2026-03-20 16:26:25.753013 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.753017 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.753021 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.753028 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.753032 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.753036 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.753040 | controller | -- \' 2026-03-20 16:26:25.753045 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.753049 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.753054 | controller | - ' logger.go:42: 16:12:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.753058 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.753062 | controller | - ' logger.go:42: 16:12:33 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.753066 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.753071 | controller | found' 2026-03-20 16:26:25.753075 | controller | - ' logger.go:42: 16:12:34 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.753079 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.753083 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.753088 | controller | - ' ' 2026-03-20 16:26:25.753092 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.753096 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.753100 | controller | - ' ' 2026-03-20 16:26:25.753105 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.753109 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.753113 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.753117 | controller | - ' exit 1' 2026-03-20 16:26:25.753121 | controller | - ' fi' 2026-03-20 16:26:25.753126 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.753130 | controller | - ' ' 2026-03-20 16:26:25.753134 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.753146 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.753150 | controller | | base64 -d)' 2026-03-20 16:26:25.753155 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.753159 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.753163 | controller | - ' exit 1' 2026-03-20 16:26:25.753170 | controller | - ' fi' 2026-03-20 16:26:25.753174 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.753180 | controller | - ' ' 2026-03-20 16:26:25.753184 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.753189 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.753193 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.753197 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.753201 | controller | --timeout=300s' 2026-03-20 16:26:25.753205 | controller | - ' ' 2026-03-20 16:26:25.753210 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.753214 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.753218 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.753222 | controller | - ' ' 2026-03-20 16:26:25.753226 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.753231 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.753235 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.753239 | controller | - ' ' 2026-03-20 16:26:25.753243 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.753249 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.753253 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.753258 | controller | available"' 2026-03-20 16:26:25.753262 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.753266 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.753270 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.753274 | controller | - ' exit 1' 2026-03-20 16:26:25.753279 | controller | - ' fi' 2026-03-20 16:26:25.753283 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.753287 | controller | - ' ' 2026-03-20 16:26:25.753291 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.753295 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.753299 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.753304 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.753308 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.753312 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.753316 | controller | -- \' 2026-03-20 16:26:25.753320 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.753324 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.753329 | controller | - ' logger.go:42: 16:12:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.753333 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.753337 | controller | - ' logger.go:42: 16:12:34 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.753341 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.753355 | controller | found' 2026-03-20 16:26:25.753359 | controller | - ' logger.go:42: 16:12:35 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.753363 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.753368 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.753372 | controller | - ' ' 2026-03-20 16:26:25.753376 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.753380 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.753384 | controller | - ' ' 2026-03-20 16:26:25.753388 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.753393 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.753397 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.753401 | controller | - ' exit 1' 2026-03-20 16:26:25.753405 | controller | - ' fi' 2026-03-20 16:26:25.753409 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.753414 | controller | - ' ' 2026-03-20 16:26:25.753418 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.753422 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.753426 | controller | | base64 -d)' 2026-03-20 16:26:25.753438 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.753442 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.753446 | controller | - ' exit 1' 2026-03-20 16:26:25.753451 | controller | - ' fi' 2026-03-20 16:26:25.753455 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.753459 | controller | - ' ' 2026-03-20 16:26:25.753463 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.753467 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.753472 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.753476 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.753480 | controller | --timeout=300s' 2026-03-20 16:26:25.753484 | controller | - ' ' 2026-03-20 16:26:25.753488 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.753493 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.753497 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.753501 | controller | - ' ' 2026-03-20 16:26:25.753505 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.753509 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.753514 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.753518 | controller | - ' ' 2026-03-20 16:26:25.753522 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.753526 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.753532 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.753537 | controller | available"' 2026-03-20 16:26:25.753541 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.753545 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.753549 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.753553 | controller | - ' exit 1' 2026-03-20 16:26:25.753561 | controller | - ' fi' 2026-03-20 16:26:25.753565 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.753569 | controller | - ' ' 2026-03-20 16:26:25.753573 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.753577 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.753582 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.753586 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.753590 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.753594 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.753598 | controller | -- \' 2026-03-20 16:26:25.753603 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.753609 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.753613 | controller | - ' logger.go:42: 16:12:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.753617 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.753621 | controller | - ' logger.go:42: 16:12:35 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.753626 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.753630 | controller | found' 2026-03-20 16:26:25.753634 | controller | - ' logger.go:42: 16:12:36 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.753638 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.753642 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.753647 | controller | - ' ' 2026-03-20 16:26:25.753651 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.753655 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.753659 | controller | - ' ' 2026-03-20 16:26:25.753663 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.753667 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.753672 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.753676 | controller | - ' exit 1' 2026-03-20 16:26:25.753680 | controller | - ' fi' 2026-03-20 16:26:25.753684 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.753689 | controller | - ' ' 2026-03-20 16:26:25.753706 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.753712 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.753716 | controller | | base64 -d)' 2026-03-20 16:26:25.753721 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.753732 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.753736 | controller | - ' exit 1' 2026-03-20 16:26:25.753740 | controller | - ' fi' 2026-03-20 16:26:25.753744 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.753749 | controller | - ' ' 2026-03-20 16:26:25.753753 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.753757 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.753762 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.753766 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.753770 | controller | --timeout=300s' 2026-03-20 16:26:25.753776 | controller | - ' ' 2026-03-20 16:26:25.753780 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.753784 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.753789 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.753793 | controller | - ' ' 2026-03-20 16:26:25.753797 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.753801 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.753805 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.753810 | controller | - ' ' 2026-03-20 16:26:25.753814 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.753818 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.753822 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.753826 | controller | available"' 2026-03-20 16:26:25.753831 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.753835 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.753839 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.753843 | controller | - ' exit 1' 2026-03-20 16:26:25.753848 | controller | - ' fi' 2026-03-20 16:26:25.753852 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.753856 | controller | - ' ' 2026-03-20 16:26:25.753860 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.753864 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.753868 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.753873 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.753877 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.753881 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.753885 | controller | -- \' 2026-03-20 16:26:25.753889 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.753894 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.753898 | controller | - ' logger.go:42: 16:12:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.753902 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.753906 | controller | - ' logger.go:42: 16:12:37 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.753910 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.753915 | controller | found' 2026-03-20 16:26:25.753919 | controller | - ' logger.go:42: 16:12:38 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.753923 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.753927 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.753931 | controller | - ' ' 2026-03-20 16:26:25.753936 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.753940 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.753944 | controller | - ' ' 2026-03-20 16:26:25.753950 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.753954 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.753958 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.753962 | controller | - ' exit 1' 2026-03-20 16:26:25.753967 | controller | - ' fi' 2026-03-20 16:26:25.753971 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.753975 | controller | - ' ' 2026-03-20 16:26:25.753981 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.753985 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.753990 | controller | | base64 -d)' 2026-03-20 16:26:25.753994 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.754005 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.754010 | controller | - ' exit 1' 2026-03-20 16:26:25.754014 | controller | - ' fi' 2026-03-20 16:26:25.754018 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.754023 | controller | - ' ' 2026-03-20 16:26:25.754027 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.754031 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.754035 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.754039 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.754043 | controller | --timeout=300s' 2026-03-20 16:26:25.754048 | controller | - ' ' 2026-03-20 16:26:25.754052 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.754056 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.754060 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.754064 | controller | - ' ' 2026-03-20 16:26:25.754068 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.754074 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.754078 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.754082 | controller | - ' ' 2026-03-20 16:26:25.754087 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.754091 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.754095 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.754099 | controller | available"' 2026-03-20 16:26:25.754103 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.754108 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.754112 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.754116 | controller | - ' exit 1' 2026-03-20 16:26:25.754120 | controller | - ' fi' 2026-03-20 16:26:25.754124 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.754128 | controller | - ' ' 2026-03-20 16:26:25.754133 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.754137 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.754141 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.754147 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.754151 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.754155 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.754159 | controller | -- \' 2026-03-20 16:26:25.754164 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.754168 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.754172 | controller | - ' logger.go:42: 16:12:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.754176 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.754182 | controller | - ' logger.go:42: 16:12:38 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.754186 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.754190 | controller | found' 2026-03-20 16:26:25.754195 | controller | - ' logger.go:42: 16:12:39 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.754199 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.754203 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.754207 | controller | - ' ' 2026-03-20 16:26:25.754211 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.754216 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.754220 | controller | - ' ' 2026-03-20 16:26:25.754226 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.754230 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.754234 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.754239 | controller | - ' exit 1' 2026-03-20 16:26:25.754243 | controller | - ' fi' 2026-03-20 16:26:25.754247 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.754251 | controller | - ' ' 2026-03-20 16:26:25.754255 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.754260 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.754264 | controller | | base64 -d)' 2026-03-20 16:26:25.754268 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.754272 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.754283 | controller | - ' exit 1' 2026-03-20 16:26:25.754288 | controller | - ' fi' 2026-03-20 16:26:25.754292 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.754297 | controller | - ' ' 2026-03-20 16:26:25.754301 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.754305 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.754309 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.754314 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.754318 | controller | --timeout=300s' 2026-03-20 16:26:25.754322 | controller | - ' ' 2026-03-20 16:26:25.754326 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.754331 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.754337 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.754341 | controller | - ' ' 2026-03-20 16:26:25.754345 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.754351 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.754355 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.754359 | controller | - ' ' 2026-03-20 16:26:25.754363 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.754368 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.754372 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.754376 | controller | available"' 2026-03-20 16:26:25.754380 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.754384 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.754389 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.754393 | controller | - ' exit 1' 2026-03-20 16:26:25.754397 | controller | - ' fi' 2026-03-20 16:26:25.754401 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.754405 | controller | - ' ' 2026-03-20 16:26:25.754410 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.754414 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.754418 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.754422 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.754426 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.754430 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.754435 | controller | -- \' 2026-03-20 16:26:25.754439 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.754443 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.754447 | controller | - ' logger.go:42: 16:12:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.754451 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.754456 | controller | - ' logger.go:42: 16:12:39 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.754460 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.754464 | controller | found' 2026-03-20 16:26:25.754468 | controller | - ' logger.go:42: 16:12:40 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.754472 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.754477 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.754481 | controller | - ' ' 2026-03-20 16:26:25.754485 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.754489 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.754493 | controller | - ' ' 2026-03-20 16:26:25.754497 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.754502 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.754506 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.754510 | controller | - ' exit 1' 2026-03-20 16:26:25.754514 | controller | - ' fi' 2026-03-20 16:26:25.754518 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.754523 | controller | - ' ' 2026-03-20 16:26:25.754527 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.754532 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.754537 | controller | | base64 -d)' 2026-03-20 16:26:25.754541 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.754545 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.754549 | controller | - ' exit 1' 2026-03-20 16:26:25.754554 | controller | - ' fi' 2026-03-20 16:26:25.754564 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.754569 | controller | - ' ' 2026-03-20 16:26:25.754573 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.754577 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.754582 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.754586 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.754590 | controller | --timeout=300s' 2026-03-20 16:26:25.754594 | controller | - ' ' 2026-03-20 16:26:25.754598 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.754603 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.754607 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.754611 | controller | - ' ' 2026-03-20 16:26:25.754615 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.754620 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.754624 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.754628 | controller | - ' ' 2026-03-20 16:26:25.754632 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.754636 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.754641 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.754645 | controller | available"' 2026-03-20 16:26:25.754649 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.754653 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.754657 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.754661 | controller | - ' exit 1' 2026-03-20 16:26:25.754666 | controller | - ' fi' 2026-03-20 16:26:25.754670 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.754674 | controller | - ' ' 2026-03-20 16:26:25.754678 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.754682 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.754687 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.754691 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.754713 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.754717 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.754721 | controller | -- \' 2026-03-20 16:26:25.754725 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.754730 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.754736 | controller | - ' logger.go:42: 16:12:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.754757 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.754761 | controller | - ' logger.go:42: 16:12:40 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.754767 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.754771 | controller | found' 2026-03-20 16:26:25.754776 | controller | - ' logger.go:42: 16:12:41 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.754780 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.754784 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.754788 | controller | - ' ' 2026-03-20 16:26:25.754793 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.754797 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.754801 | controller | - ' ' 2026-03-20 16:26:25.754805 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.754809 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.754813 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.754818 | controller | - ' exit 1' 2026-03-20 16:26:25.754822 | controller | - ' fi' 2026-03-20 16:26:25.754826 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.754830 | controller | - ' ' 2026-03-20 16:26:25.754834 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.754838 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.754843 | controller | | base64 -d)' 2026-03-20 16:26:25.754847 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.754851 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.754855 | controller | - ' exit 1' 2026-03-20 16:26:25.754859 | controller | - ' fi' 2026-03-20 16:26:25.754864 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.754877 | controller | - ' ' 2026-03-20 16:26:25.754882 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.754886 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.754890 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.754895 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.754899 | controller | --timeout=300s' 2026-03-20 16:26:25.754903 | controller | - ' ' 2026-03-20 16:26:25.754907 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.754911 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.754916 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.754920 | controller | - ' ' 2026-03-20 16:26:25.754924 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.754928 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.754932 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.754937 | controller | - ' ' 2026-03-20 16:26:25.754941 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.754945 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.754952 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.754956 | controller | available"' 2026-03-20 16:26:25.754961 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.754965 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.754969 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.754973 | controller | - ' exit 1' 2026-03-20 16:26:25.754977 | controller | - ' fi' 2026-03-20 16:26:25.754981 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.754986 | controller | - ' ' 2026-03-20 16:26:25.754990 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.754994 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.754998 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.755002 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.755006 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.755011 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.755015 | controller | -- \' 2026-03-20 16:26:25.755019 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.755023 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.755027 | controller | - ' logger.go:42: 16:12:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.755031 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.755036 | controller | - ' logger.go:42: 16:12:41 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.755040 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.755044 | controller | found' 2026-03-20 16:26:25.755048 | controller | - ' logger.go:42: 16:12:42 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.755053 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.755057 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.755061 | controller | - ' ' 2026-03-20 16:26:25.755065 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.755069 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.755074 | controller | - ' ' 2026-03-20 16:26:25.755078 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.755082 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.755086 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.755090 | controller | - ' exit 1' 2026-03-20 16:26:25.755095 | controller | - ' fi' 2026-03-20 16:26:25.755099 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.755103 | controller | - ' ' 2026-03-20 16:26:25.755107 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.755111 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.755115 | controller | | base64 -d)' 2026-03-20 16:26:25.755120 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.755124 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.755128 | controller | - ' exit 1' 2026-03-20 16:26:25.755132 | controller | - ' fi' 2026-03-20 16:26:25.755137 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.755142 | controller | - ' ' 2026-03-20 16:26:25.755155 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.755160 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.755164 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.755169 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.755173 | controller | --timeout=300s' 2026-03-20 16:26:25.755177 | controller | - ' ' 2026-03-20 16:26:25.755182 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.755186 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.755190 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.755194 | controller | - ' ' 2026-03-20 16:26:25.755198 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.755202 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.755207 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.755211 | controller | - ' ' 2026-03-20 16:26:25.755215 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.755219 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.755223 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.755228 | controller | available"' 2026-03-20 16:26:25.755233 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.755238 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.755243 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.755247 | controller | - ' exit 1' 2026-03-20 16:26:25.755252 | controller | - ' fi' 2026-03-20 16:26:25.755256 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.755260 | controller | - ' ' 2026-03-20 16:26:25.755264 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.755269 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.755273 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.755277 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.755283 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.755287 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.755291 | controller | -- \' 2026-03-20 16:26:25.755295 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.755300 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.755304 | controller | - ' logger.go:42: 16:12:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.755308 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.755312 | controller | - ' logger.go:42: 16:12:43 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.755316 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.755321 | controller | found' 2026-03-20 16:26:25.755326 | controller | - ' logger.go:42: 16:12:44 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.755332 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.755337 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.755341 | controller | - ' ' 2026-03-20 16:26:25.755345 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.755349 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.755354 | controller | - ' ' 2026-03-20 16:26:25.755358 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.755362 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.755366 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.755371 | controller | - ' exit 1' 2026-03-20 16:26:25.755375 | controller | - ' fi' 2026-03-20 16:26:25.755379 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.755383 | controller | - ' ' 2026-03-20 16:26:25.755387 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.755392 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.755396 | controller | | base64 -d)' 2026-03-20 16:26:25.755400 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.755404 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.755409 | controller | - ' exit 1' 2026-03-20 16:26:25.755413 | controller | - ' fi' 2026-03-20 16:26:25.755417 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.755421 | controller | - ' ' 2026-03-20 16:26:25.755425 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.755436 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.755441 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.755445 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.755449 | controller | --timeout=300s' 2026-03-20 16:26:25.755453 | controller | - ' ' 2026-03-20 16:26:25.755458 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.755462 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.755466 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.755470 | controller | - ' ' 2026-03-20 16:26:25.755474 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.755479 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.755483 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.755487 | controller | - ' ' 2026-03-20 16:26:25.755491 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.755496 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.755500 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.755504 | controller | available"' 2026-03-20 16:26:25.755508 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.755513 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.755517 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.755521 | controller | - ' exit 1' 2026-03-20 16:26:25.755525 | controller | - ' fi' 2026-03-20 16:26:25.755529 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.755535 | controller | - ' ' 2026-03-20 16:26:25.755540 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.755544 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.755548 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.755552 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.755557 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.755561 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.755565 | controller | -- \' 2026-03-20 16:26:25.755569 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.755573 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.755578 | controller | - ' logger.go:42: 16:12:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.755582 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.755586 | controller | - ' logger.go:42: 16:12:44 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.755590 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.755595 | controller | found' 2026-03-20 16:26:25.755599 | controller | - ' logger.go:42: 16:12:45 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.755603 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.755607 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.755612 | controller | - ' ' 2026-03-20 16:26:25.755616 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.755620 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.755624 | controller | - ' ' 2026-03-20 16:26:25.755628 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.755633 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.755637 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.755641 | controller | - ' exit 1' 2026-03-20 16:26:25.755645 | controller | - ' fi' 2026-03-20 16:26:25.755650 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.755654 | controller | - ' ' 2026-03-20 16:26:25.755676 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.755680 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.755685 | controller | | base64 -d)' 2026-03-20 16:26:25.755689 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.755704 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.755710 | controller | - ' exit 1' 2026-03-20 16:26:25.755714 | controller | - ' fi' 2026-03-20 16:26:25.755719 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.755723 | controller | - ' ' 2026-03-20 16:26:25.755727 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.755731 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.755743 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.755747 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.755752 | controller | --timeout=300s' 2026-03-20 16:26:25.755756 | controller | - ' ' 2026-03-20 16:26:25.755760 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.755766 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.755771 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.755775 | controller | - ' ' 2026-03-20 16:26:25.755779 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.755783 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.755787 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.755792 | controller | - ' ' 2026-03-20 16:26:25.755796 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.755800 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.755804 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.755809 | controller | available"' 2026-03-20 16:26:25.755813 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.755817 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.755821 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.755825 | controller | - ' exit 1' 2026-03-20 16:26:25.755830 | controller | - ' fi' 2026-03-20 16:26:25.755834 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.755838 | controller | - ' ' 2026-03-20 16:26:25.755842 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.755846 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.755851 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.755855 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.755859 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.755863 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.755867 | controller | -- \' 2026-03-20 16:26:25.755872 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.755876 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.755880 | controller | - ' logger.go:42: 16:12:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.755884 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.755888 | controller | - ' logger.go:42: 16:12:45 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.755893 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.755897 | controller | found' 2026-03-20 16:26:25.755901 | controller | - ' logger.go:42: 16:12:46 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.755905 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.755909 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.755914 | controller | - ' ' 2026-03-20 16:26:25.755920 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.755924 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.755928 | controller | - ' ' 2026-03-20 16:26:25.755933 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.755938 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.755943 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.755947 | controller | - ' exit 1' 2026-03-20 16:26:25.755951 | controller | - ' fi' 2026-03-20 16:26:25.755955 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.755959 | controller | - ' ' 2026-03-20 16:26:25.755964 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.755968 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.755972 | controller | | base64 -d)' 2026-03-20 16:26:25.755976 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.755981 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.755985 | controller | - ' exit 1' 2026-03-20 16:26:25.755989 | controller | - ' fi' 2026-03-20 16:26:25.755993 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.755997 | controller | - ' ' 2026-03-20 16:26:25.756001 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.756006 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.756017 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.756022 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.756026 | controller | --timeout=300s' 2026-03-20 16:26:25.756030 | controller | - ' ' 2026-03-20 16:26:25.756035 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.756039 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.756043 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.756047 | controller | - ' ' 2026-03-20 16:26:25.756051 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.756056 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.756060 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.756064 | controller | - ' ' 2026-03-20 16:26:25.756068 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.756072 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.756076 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.756081 | controller | available"' 2026-03-20 16:26:25.756085 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.756089 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.756093 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.756097 | controller | - ' exit 1' 2026-03-20 16:26:25.756102 | controller | - ' fi' 2026-03-20 16:26:25.756106 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.756110 | controller | - ' ' 2026-03-20 16:26:25.756114 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.756118 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.756122 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.756127 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.756134 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.756138 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.756142 | controller | -- \' 2026-03-20 16:26:25.756146 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.756151 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.756155 | controller | - ' logger.go:42: 16:12:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.756159 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.756173 | controller | - ' logger.go:42: 16:12:46 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.756177 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.756182 | controller | found' 2026-03-20 16:26:25.756186 | controller | - ' logger.go:42: 16:12:47 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.756190 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.756195 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.756199 | controller | - ' ' 2026-03-20 16:26:25.756203 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.756207 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.756211 | controller | - ' ' 2026-03-20 16:26:25.756216 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.756220 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.756224 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.756230 | controller | - ' exit 1' 2026-03-20 16:26:25.756234 | controller | - ' fi' 2026-03-20 16:26:25.756238 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.756243 | controller | - ' ' 2026-03-20 16:26:25.756247 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.756251 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.756255 | controller | | base64 -d)' 2026-03-20 16:26:25.756259 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.756264 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.756268 | controller | - ' exit 1' 2026-03-20 16:26:25.756272 | controller | - ' fi' 2026-03-20 16:26:25.756276 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.756280 | controller | - ' ' 2026-03-20 16:26:25.756288 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.756292 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.756297 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.756308 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.756312 | controller | --timeout=300s' 2026-03-20 16:26:25.756317 | controller | - ' ' 2026-03-20 16:26:25.756321 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.756325 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.756330 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.756334 | controller | - ' ' 2026-03-20 16:26:25.756338 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.756346 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.756350 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.756354 | controller | - ' ' 2026-03-20 16:26:25.756359 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.756363 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.756368 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.756374 | controller | available"' 2026-03-20 16:26:25.756379 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.756385 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.756390 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.756394 | controller | - ' exit 1' 2026-03-20 16:26:25.756398 | controller | - ' fi' 2026-03-20 16:26:25.756403 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.756407 | controller | - ' ' 2026-03-20 16:26:25.756411 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.756415 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.756422 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.756426 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.756430 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.756434 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.756439 | controller | -- \' 2026-03-20 16:26:25.756443 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.756447 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.756451 | controller | - ' logger.go:42: 16:12:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.756455 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.756460 | controller | - ' logger.go:42: 16:12:47 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.756464 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.756468 | controller | found' 2026-03-20 16:26:25.756472 | controller | - ' logger.go:42: 16:12:48 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.756476 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.756480 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.756485 | controller | - ' ' 2026-03-20 16:26:25.756489 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.756495 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.756499 | controller | - ' ' 2026-03-20 16:26:25.756503 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.756507 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.756511 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.756515 | controller | - ' exit 1' 2026-03-20 16:26:25.756520 | controller | - ' fi' 2026-03-20 16:26:25.756524 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.756530 | controller | - ' ' 2026-03-20 16:26:25.756537 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.756543 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.756551 | controller | | base64 -d)' 2026-03-20 16:26:25.756557 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.756563 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.756569 | controller | - ' exit 1' 2026-03-20 16:26:25.756574 | controller | - ' fi' 2026-03-20 16:26:25.756579 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.756583 | controller | - ' ' 2026-03-20 16:26:25.756588 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.756592 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.756596 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.756611 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.756616 | controller | --timeout=300s' 2026-03-20 16:26:25.756620 | controller | - ' ' 2026-03-20 16:26:25.756624 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.756629 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.756633 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.756637 | controller | - ' ' 2026-03-20 16:26:25.756641 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.756645 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.756650 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.756654 | controller | - ' ' 2026-03-20 16:26:25.756658 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.756662 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.756667 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.756671 | controller | available"' 2026-03-20 16:26:25.756675 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.756679 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.756684 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.756688 | controller | - ' exit 1' 2026-03-20 16:26:25.756708 | controller | - ' fi' 2026-03-20 16:26:25.756715 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.756719 | controller | - ' ' 2026-03-20 16:26:25.756723 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.756728 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.756732 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.756736 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.756740 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.756744 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.756749 | controller | -- \' 2026-03-20 16:26:25.756753 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.756757 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.756763 | controller | - ' logger.go:42: 16:12:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.756768 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.756772 | controller | - ' logger.go:42: 16:12:49 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.756776 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.756780 | controller | found' 2026-03-20 16:26:25.756784 | controller | - ' logger.go:42: 16:12:50 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.756789 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.756793 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.756797 | controller | - ' ' 2026-03-20 16:26:25.756801 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.756805 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.756809 | controller | - ' ' 2026-03-20 16:26:25.756814 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.756818 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.756822 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.756826 | controller | - ' exit 1' 2026-03-20 16:26:25.756831 | controller | - ' fi' 2026-03-20 16:26:25.756835 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.756839 | controller | - ' ' 2026-03-20 16:26:25.756843 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.756847 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.756852 | controller | | base64 -d)' 2026-03-20 16:26:25.756856 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.756860 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.756864 | controller | - ' exit 1' 2026-03-20 16:26:25.756868 | controller | - ' fi' 2026-03-20 16:26:25.756873 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.756877 | controller | - ' ' 2026-03-20 16:26:25.756881 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.756885 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.756889 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.756894 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.756898 | controller | --timeout=300s' 2026-03-20 16:26:25.756902 | controller | - ' ' 2026-03-20 16:26:25.756914 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.756919 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.756924 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.756928 | controller | - ' ' 2026-03-20 16:26:25.756932 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.756936 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.756940 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.756945 | controller | - ' ' 2026-03-20 16:26:25.756949 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.756953 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.756957 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.756963 | controller | available"' 2026-03-20 16:26:25.756967 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.756972 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.756976 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.756980 | controller | - ' exit 1' 2026-03-20 16:26:25.756984 | controller | - ' fi' 2026-03-20 16:26:25.756988 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.756993 | controller | - ' ' 2026-03-20 16:26:25.756997 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.757001 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.757005 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.757009 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.757013 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.757018 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.757022 | controller | -- \' 2026-03-20 16:26:25.757026 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.757030 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.757035 | controller | - ' logger.go:42: 16:12:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.757039 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.757043 | controller | - ' logger.go:42: 16:12:50 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.757047 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.757051 | controller | found' 2026-03-20 16:26:25.757055 | controller | - ' logger.go:42: 16:12:51 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.757060 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.757064 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.757068 | controller | - ' ' 2026-03-20 16:26:25.757072 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.757076 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.757081 | controller | - ' ' 2026-03-20 16:26:25.757087 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.757091 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.757096 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.757100 | controller | - ' exit 1' 2026-03-20 16:26:25.757104 | controller | - ' fi' 2026-03-20 16:26:25.757108 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.757113 | controller | - ' ' 2026-03-20 16:26:25.757117 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.757121 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.757125 | controller | | base64 -d)' 2026-03-20 16:26:25.757129 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.757134 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.757138 | controller | - ' exit 1' 2026-03-20 16:26:25.757142 | controller | - ' fi' 2026-03-20 16:26:25.757146 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.757150 | controller | - ' ' 2026-03-20 16:26:25.757154 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.757160 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.757165 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.757169 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.757173 | controller | --timeout=300s' 2026-03-20 16:26:25.757177 | controller | - ' ' 2026-03-20 16:26:25.757189 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.757195 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.757199 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.757203 | controller | - ' ' 2026-03-20 16:26:25.757207 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.757211 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.757216 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.757220 | controller | - ' ' 2026-03-20 16:26:25.757224 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.757228 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.757233 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.757237 | controller | available"' 2026-03-20 16:26:25.757241 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.757245 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.757250 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.757254 | controller | - ' exit 1' 2026-03-20 16:26:25.757258 | controller | - ' fi' 2026-03-20 16:26:25.757262 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.757266 | controller | - ' ' 2026-03-20 16:26:25.757271 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.757275 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.757279 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.757283 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.757288 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.757292 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.757298 | controller | -- \' 2026-03-20 16:26:25.757302 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.757306 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.757311 | controller | - ' logger.go:42: 16:12:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.757315 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.757319 | controller | - ' logger.go:42: 16:12:52 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.757323 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.757327 | controller | found' 2026-03-20 16:26:25.757331 | controller | - ' logger.go:42: 16:12:53 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.757338 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.757342 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.757346 | controller | - ' ' 2026-03-20 16:26:25.757351 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.757355 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.757359 | controller | - ' ' 2026-03-20 16:26:25.757363 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.757367 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.757372 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.757376 | controller | - ' exit 1' 2026-03-20 16:26:25.757380 | controller | - ' fi' 2026-03-20 16:26:25.757386 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.757390 | controller | - ' ' 2026-03-20 16:26:25.757395 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.757399 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.757403 | controller | | base64 -d)' 2026-03-20 16:26:25.757407 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.757411 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.757415 | controller | - ' exit 1' 2026-03-20 16:26:25.757420 | controller | - ' fi' 2026-03-20 16:26:25.757424 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.757428 | controller | - ' ' 2026-03-20 16:26:25.757432 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.757436 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.757440 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.757445 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.757449 | controller | --timeout=300s' 2026-03-20 16:26:25.757453 | controller | - ' ' 2026-03-20 16:26:25.757457 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.757469 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.757474 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.757478 | controller | - ' ' 2026-03-20 16:26:25.757482 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.757486 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.757491 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.757495 | controller | - ' ' 2026-03-20 16:26:25.757499 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.757503 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.757509 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.757514 | controller | available"' 2026-03-20 16:26:25.757518 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.757522 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.757526 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.757530 | controller | - ' exit 1' 2026-03-20 16:26:25.757535 | controller | - ' fi' 2026-03-20 16:26:25.757539 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.757548 | controller | - ' ' 2026-03-20 16:26:25.757552 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.757556 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.757560 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.757564 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.757569 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.757573 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.757577 | controller | -- \' 2026-03-20 16:26:25.757581 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.757585 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.757590 | controller | - ' logger.go:42: 16:12:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.757594 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.757598 | controller | - ' logger.go:42: 16:12:53 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.757602 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.757606 | controller | found' 2026-03-20 16:26:25.757611 | controller | - ' logger.go:42: 16:12:54 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.757615 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.757619 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.757623 | controller | - ' ' 2026-03-20 16:26:25.757627 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.757632 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.757636 | controller | - ' ' 2026-03-20 16:26:25.757640 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.757644 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.757648 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.757653 | controller | - ' exit 1' 2026-03-20 16:26:25.757657 | controller | - ' fi' 2026-03-20 16:26:25.757661 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.757667 | controller | - ' ' 2026-03-20 16:26:25.757671 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.757675 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.757680 | controller | | base64 -d)' 2026-03-20 16:26:25.757684 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.757688 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.757721 | controller | - ' exit 1' 2026-03-20 16:26:25.757750 | controller | - ' fi' 2026-03-20 16:26:25.757754 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.757758 | controller | - ' ' 2026-03-20 16:26:25.757763 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.757767 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.757771 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.757775 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.757779 | controller | --timeout=300s' 2026-03-20 16:26:25.757784 | controller | - ' ' 2026-03-20 16:26:25.757788 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.757802 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.757807 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.757811 | controller | - ' ' 2026-03-20 16:26:25.757815 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.757819 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.757823 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.757828 | controller | - ' ' 2026-03-20 16:26:25.757832 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.757836 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.757840 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.757845 | controller | available"' 2026-03-20 16:26:25.757849 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.757855 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.757860 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.757864 | controller | - ' exit 1' 2026-03-20 16:26:25.757868 | controller | - ' fi' 2026-03-20 16:26:25.757872 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.757876 | controller | - ' ' 2026-03-20 16:26:25.757881 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.757885 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.757889 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.757893 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.757897 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.757901 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.757906 | controller | -- \' 2026-03-20 16:26:25.757910 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.757914 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.757918 | controller | - ' logger.go:42: 16:12:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.757922 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.757927 | controller | - ' logger.go:42: 16:12:54 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.757931 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.757935 | controller | found' 2026-03-20 16:26:25.757939 | controller | - ' logger.go:42: 16:12:55 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.757943 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.757948 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.757952 | controller | - ' ' 2026-03-20 16:26:25.757956 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.757960 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.757964 | controller | - ' ' 2026-03-20 16:26:25.757968 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.757972 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.757978 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.757983 | controller | - ' exit 1' 2026-03-20 16:26:25.757987 | controller | - ' fi' 2026-03-20 16:26:25.757991 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.757995 | controller | - ' ' 2026-03-20 16:26:25.757999 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.758004 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.758008 | controller | | base64 -d)' 2026-03-20 16:26:25.758012 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.758016 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.758020 | controller | - ' exit 1' 2026-03-20 16:26:25.758025 | controller | - ' fi' 2026-03-20 16:26:25.758029 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.758033 | controller | - ' ' 2026-03-20 16:26:25.758037 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.758041 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.758046 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.758050 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.758054 | controller | --timeout=300s' 2026-03-20 16:26:25.758058 | controller | - ' ' 2026-03-20 16:26:25.758062 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.758075 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.758080 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.758084 | controller | - ' ' 2026-03-20 16:26:25.758088 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.758092 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.758097 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.758101 | controller | - ' ' 2026-03-20 16:26:25.758105 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.758109 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.758113 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.758118 | controller | available"' 2026-03-20 16:26:25.758122 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.758126 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.758130 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.758134 | controller | - ' exit 1' 2026-03-20 16:26:25.758139 | controller | - ' fi' 2026-03-20 16:26:25.758143 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.758147 | controller | - ' ' 2026-03-20 16:26:25.758151 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.758155 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.758159 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.758164 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.758168 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.758174 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.758178 | controller | -- \' 2026-03-20 16:26:25.758182 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.758187 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.758191 | controller | - ' logger.go:42: 16:12:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.758195 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.758199 | controller | - ' logger.go:42: 16:12:55 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.758203 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.758207 | controller | found' 2026-03-20 16:26:25.758211 | controller | - ' logger.go:42: 16:12:56 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.758216 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.758220 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.758224 | controller | - ' ' 2026-03-20 16:26:25.758228 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.758232 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.758237 | controller | - ' ' 2026-03-20 16:26:25.758241 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.758245 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.758249 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.758253 | controller | - ' exit 1' 2026-03-20 16:26:25.758258 | controller | - ' fi' 2026-03-20 16:26:25.758262 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.758266 | controller | - ' ' 2026-03-20 16:26:25.758270 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.758274 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.758279 | controller | | base64 -d)' 2026-03-20 16:26:25.758283 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.758287 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.758291 | controller | - ' exit 1' 2026-03-20 16:26:25.758295 | controller | - ' fi' 2026-03-20 16:26:25.758300 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.758304 | controller | - ' ' 2026-03-20 16:26:25.758308 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.758312 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.758338 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.758343 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.758348 | controller | --timeout=300s' 2026-03-20 16:26:25.758352 | controller | - ' ' 2026-03-20 16:26:25.758356 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.758360 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.758375 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.758379 | controller | - ' ' 2026-03-20 16:26:25.758384 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.758388 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.758396 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.758400 | controller | - ' ' 2026-03-20 16:26:25.758404 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.758408 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.758413 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.758417 | controller | available"' 2026-03-20 16:26:25.758421 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.758425 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.758429 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.758434 | controller | - ' exit 1' 2026-03-20 16:26:25.758438 | controller | - ' fi' 2026-03-20 16:26:25.758444 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.758448 | controller | - ' ' 2026-03-20 16:26:25.758452 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.758456 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.758461 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.758465 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.758469 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.758473 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.758477 | controller | -- \' 2026-03-20 16:26:25.758481 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.758486 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.758490 | controller | - ' logger.go:42: 16:12:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.758494 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.758498 | controller | - ' logger.go:42: 16:12:56 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.758502 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.758507 | controller | found' 2026-03-20 16:26:25.758511 | controller | - ' logger.go:42: 16:12:57 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.758515 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.758519 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.758523 | controller | - ' ' 2026-03-20 16:26:25.758528 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.758532 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.758536 | controller | - ' ' 2026-03-20 16:26:25.758540 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.758544 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.758548 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.758553 | controller | - ' exit 1' 2026-03-20 16:26:25.758557 | controller | - ' fi' 2026-03-20 16:26:25.758561 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.758565 | controller | - ' ' 2026-03-20 16:26:25.758570 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.758574 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.758579 | controller | | base64 -d)' 2026-03-20 16:26:25.758584 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.758588 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.758592 | controller | - ' exit 1' 2026-03-20 16:26:25.758596 | controller | - ' fi' 2026-03-20 16:26:25.758600 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.758605 | controller | - ' ' 2026-03-20 16:26:25.758609 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.758614 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.758620 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.758625 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.758631 | controller | --timeout=300s' 2026-03-20 16:26:25.758636 | controller | - ' ' 2026-03-20 16:26:25.758641 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.758646 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.758661 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.758667 | controller | - ' ' 2026-03-20 16:26:25.758672 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.758678 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.758683 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.758689 | controller | - ' ' 2026-03-20 16:26:25.758712 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.758718 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.758724 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.758729 | controller | available"' 2026-03-20 16:26:25.758735 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.758741 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.758747 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.758753 | controller | - ' exit 1' 2026-03-20 16:26:25.758759 | controller | - ' fi' 2026-03-20 16:26:25.758764 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.758770 | controller | - ' ' 2026-03-20 16:26:25.758776 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.758782 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.758789 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.758795 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.758801 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.758806 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.758812 | controller | -- \' 2026-03-20 16:26:25.758817 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.758823 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.758829 | controller | - ' logger.go:42: 16:12:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.758838 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.758843 | controller | - ' logger.go:42: 16:12:57 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.758849 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.758855 | controller | found' 2026-03-20 16:26:25.758863 | controller | - ' logger.go:42: 16:12:58 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.758869 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.758875 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.758881 | controller | - ' ' 2026-03-20 16:26:25.758889 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.758895 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.758901 | controller | - ' ' 2026-03-20 16:26:25.758907 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.758913 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.758919 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.758926 | controller | - ' exit 1' 2026-03-20 16:26:25.758931 | controller | - ' fi' 2026-03-20 16:26:25.758937 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.758943 | controller | - ' ' 2026-03-20 16:26:25.758949 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.758955 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.758961 | controller | | base64 -d)' 2026-03-20 16:26:25.758967 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.758972 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.758978 | controller | - ' exit 1' 2026-03-20 16:26:25.758984 | controller | - ' fi' 2026-03-20 16:26:25.758990 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.758999 | controller | - ' ' 2026-03-20 16:26:25.759006 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.759012 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.759018 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.759024 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.759030 | controller | --timeout=300s' 2026-03-20 16:26:25.759037 | controller | - ' ' 2026-03-20 16:26:25.759046 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.759052 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.759057 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.759063 | controller | - ' ' 2026-03-20 16:26:25.759081 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.759088 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.759094 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.759100 | controller | - ' ' 2026-03-20 16:26:25.759105 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.759111 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.759117 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.759127 | controller | available"' 2026-03-20 16:26:25.759132 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.759139 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.759144 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.759150 | controller | - ' exit 1' 2026-03-20 16:26:25.759155 | controller | - ' fi' 2026-03-20 16:26:25.759161 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.759167 | controller | - ' ' 2026-03-20 16:26:25.759173 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.759179 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.759185 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.759191 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.759198 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.759204 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.759209 | controller | -- \' 2026-03-20 16:26:25.759216 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.759222 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.759228 | controller | - ' logger.go:42: 16:12:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.759234 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.759240 | controller | - ' logger.go:42: 16:12:58 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.759246 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.759252 | controller | found' 2026-03-20 16:26:25.759258 | controller | - ' logger.go:42: 16:12:59 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.759264 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.759270 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.759276 | controller | - ' ' 2026-03-20 16:26:25.759282 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.759288 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.759293 | controller | - ' ' 2026-03-20 16:26:25.759299 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.759304 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.759310 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.759315 | controller | - ' exit 1' 2026-03-20 16:26:25.759321 | controller | - ' fi' 2026-03-20 16:26:25.759326 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.759331 | controller | - ' ' 2026-03-20 16:26:25.759340 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.759346 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.759351 | controller | | base64 -d)' 2026-03-20 16:26:25.759357 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.759362 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.759368 | controller | - ' exit 1' 2026-03-20 16:26:25.759373 | controller | - ' fi' 2026-03-20 16:26:25.759379 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.759384 | controller | - ' ' 2026-03-20 16:26:25.759389 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.759398 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.759403 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.759409 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.759414 | controller | --timeout=300s' 2026-03-20 16:26:25.759421 | controller | - ' ' 2026-03-20 16:26:25.759425 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.759429 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.759433 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.759438 | controller | - ' ' 2026-03-20 16:26:25.759451 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.759456 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.759460 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.759464 | controller | - ' ' 2026-03-20 16:26:25.759468 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.759473 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.759477 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.759481 | controller | available"' 2026-03-20 16:26:25.759485 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.759489 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.759494 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.759498 | controller | - ' exit 1' 2026-03-20 16:26:25.759502 | controller | - ' fi' 2026-03-20 16:26:25.759506 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.759511 | controller | - ' ' 2026-03-20 16:26:25.759515 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.759519 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.759523 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.759527 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.759531 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.759535 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.759540 | controller | -- \' 2026-03-20 16:26:25.759544 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.759548 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.759552 | controller | - ' logger.go:42: 16:12:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.759557 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.759561 | controller | - ' logger.go:42: 16:13:00 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.759565 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.759569 | controller | found' 2026-03-20 16:26:25.759573 | controller | - ' logger.go:42: 16:13:01 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.759578 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.759584 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.759588 | controller | - ' ' 2026-03-20 16:26:25.759592 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.759597 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.759601 | controller | - ' ' 2026-03-20 16:26:25.759605 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.759609 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.759614 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.759618 | controller | - ' exit 1' 2026-03-20 16:26:25.759622 | controller | - ' fi' 2026-03-20 16:26:25.759626 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.759631 | controller | - ' ' 2026-03-20 16:26:25.759635 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.759639 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.759643 | controller | | base64 -d)' 2026-03-20 16:26:25.759647 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.759652 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.759659 | controller | - ' exit 1' 2026-03-20 16:26:25.759710 | controller | - ' fi' 2026-03-20 16:26:25.759720 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.759725 | controller | - ' ' 2026-03-20 16:26:25.759730 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.759735 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.759741 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.759746 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.759751 | controller | --timeout=300s' 2026-03-20 16:26:25.759756 | controller | - ' ' 2026-03-20 16:26:25.759761 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.759766 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.759773 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.759779 | controller | - ' ' 2026-03-20 16:26:25.759784 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.759799 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.759805 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.759810 | controller | - ' ' 2026-03-20 16:26:25.759815 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.759820 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.759825 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.759830 | controller | available"' 2026-03-20 16:26:25.759835 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.759840 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.759845 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.759850 | controller | - ' exit 1' 2026-03-20 16:26:25.759855 | controller | - ' fi' 2026-03-20 16:26:25.759860 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.759865 | controller | - ' ' 2026-03-20 16:26:25.759873 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.759878 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.759883 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.759888 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.759893 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.759898 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.759903 | controller | -- \' 2026-03-20 16:26:25.759908 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.759913 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.759918 | controller | - ' logger.go:42: 16:13:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.759924 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.759928 | controller | - ' logger.go:42: 16:13:01 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.759932 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.759937 | controller | found' 2026-03-20 16:26:25.759941 | controller | - ' logger.go:42: 16:13:02 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.759945 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.759949 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.759956 | controller | - ' ' 2026-03-20 16:26:25.759960 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.759965 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.759969 | controller | - ' ' 2026-03-20 16:26:25.759973 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.759977 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.759982 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.759986 | controller | - ' exit 1' 2026-03-20 16:26:25.759990 | controller | - ' fi' 2026-03-20 16:26:25.759995 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.759999 | controller | - ' ' 2026-03-20 16:26:25.760003 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.760007 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.760011 | controller | | base64 -d)' 2026-03-20 16:26:25.760016 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.760020 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.760024 | controller | - ' exit 1' 2026-03-20 16:26:25.760028 | controller | - ' fi' 2026-03-20 16:26:25.760033 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.760037 | controller | - ' ' 2026-03-20 16:26:25.760041 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.760045 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.760049 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.760053 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.760058 | controller | --timeout=300s' 2026-03-20 16:26:25.760062 | controller | - ' ' 2026-03-20 16:26:25.760066 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.760070 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.760077 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.760081 | controller | - ' ' 2026-03-20 16:26:25.760085 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.760097 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.760102 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.760106 | controller | - ' ' 2026-03-20 16:26:25.760111 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.760115 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.760119 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.760123 | controller | available"' 2026-03-20 16:26:25.760127 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.760132 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.760136 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.760140 | controller | - ' exit 1' 2026-03-20 16:26:25.760144 | controller | - ' fi' 2026-03-20 16:26:25.760149 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.760153 | controller | - ' ' 2026-03-20 16:26:25.760157 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.760161 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.760166 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.760170 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.760174 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.760178 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.760183 | controller | -- \' 2026-03-20 16:26:25.760187 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.760191 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.760195 | controller | - ' logger.go:42: 16:13:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.760199 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.760204 | controller | - ' logger.go:42: 16:13:02 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.760208 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.760212 | controller | found' 2026-03-20 16:26:25.760216 | controller | - ' logger.go:42: 16:13:03 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.760220 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.760225 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.760229 | controller | - ' ' 2026-03-20 16:26:25.760233 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.760237 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.760242 | controller | - ' ' 2026-03-20 16:26:25.760246 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.760250 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.760255 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.760261 | controller | - ' exit 1' 2026-03-20 16:26:25.760265 | controller | - ' fi' 2026-03-20 16:26:25.760269 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.760273 | controller | - ' ' 2026-03-20 16:26:25.760278 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.760282 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.760286 | controller | | base64 -d)' 2026-03-20 16:26:25.760291 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.760295 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.760299 | controller | - ' exit 1' 2026-03-20 16:26:25.760303 | controller | - ' fi' 2026-03-20 16:26:25.760308 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.760312 | controller | - ' ' 2026-03-20 16:26:25.760316 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.760321 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.760325 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.760329 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.760333 | controller | --timeout=300s' 2026-03-20 16:26:25.760337 | controller | - ' ' 2026-03-20 16:26:25.760342 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.760346 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.760350 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.760355 | controller | - ' ' 2026-03-20 16:26:25.760359 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.760363 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.760375 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.760379 | controller | - ' ' 2026-03-20 16:26:25.760384 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.760388 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.760394 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.760399 | controller | available"' 2026-03-20 16:26:25.760403 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.760407 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.760412 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.760416 | controller | - ' exit 1' 2026-03-20 16:26:25.760420 | controller | - ' fi' 2026-03-20 16:26:25.760424 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.760429 | controller | - ' ' 2026-03-20 16:26:25.760433 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.760437 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.760441 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.760446 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.760450 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.760456 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.760460 | controller | -- \' 2026-03-20 16:26:25.760464 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.760469 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.760473 | controller | - ' logger.go:42: 16:13:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.760477 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.760482 | controller | - ' logger.go:42: 16:13:03 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.760486 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.760490 | controller | found' 2026-03-20 16:26:25.760494 | controller | - ' logger.go:42: 16:13:04 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.760499 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.760503 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.760507 | controller | - ' ' 2026-03-20 16:26:25.760512 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.760516 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.760520 | controller | - ' ' 2026-03-20 16:26:25.760524 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.760529 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.760533 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.760539 | controller | - ' exit 1' 2026-03-20 16:26:25.760543 | controller | - ' fi' 2026-03-20 16:26:25.760548 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.760552 | controller | - ' ' 2026-03-20 16:26:25.760556 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.760561 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.760565 | controller | | base64 -d)' 2026-03-20 16:26:25.760569 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.760573 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.760577 | controller | - ' exit 1' 2026-03-20 16:26:25.760582 | controller | - ' fi' 2026-03-20 16:26:25.760586 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.760590 | controller | - ' ' 2026-03-20 16:26:25.760594 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.760599 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.760603 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.760607 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.760611 | controller | --timeout=300s' 2026-03-20 16:26:25.760616 | controller | - ' ' 2026-03-20 16:26:25.760620 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.760624 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.760628 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.760633 | controller | - ' ' 2026-03-20 16:26:25.760637 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.760641 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.760655 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.760660 | controller | - ' ' 2026-03-20 16:26:25.760664 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.760671 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.760675 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.760679 | controller | available"' 2026-03-20 16:26:25.760684 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.760688 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.760703 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.760709 | controller | - ' exit 1' 2026-03-20 16:26:25.760714 | controller | - ' fi' 2026-03-20 16:26:25.760718 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.760725 | controller | - ' ' 2026-03-20 16:26:25.760729 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.760733 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.760737 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.760742 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.760746 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.760750 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.760755 | controller | -- \' 2026-03-20 16:26:25.760759 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.760763 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.760767 | controller | - ' logger.go:42: 16:13:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.760771 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.760776 | controller | - ' logger.go:42: 16:13:04 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.760780 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.760784 | controller | found' 2026-03-20 16:26:25.760788 | controller | - ' logger.go:42: 16:13:05 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.760793 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.760797 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.760801 | controller | - ' ' 2026-03-20 16:26:25.760806 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.760810 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.760814 | controller | - ' ' 2026-03-20 16:26:25.760818 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.760823 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.760827 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.760831 | controller | - ' exit 1' 2026-03-20 16:26:25.760835 | controller | - ' fi' 2026-03-20 16:26:25.760840 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.760844 | controller | - ' ' 2026-03-20 16:26:25.760848 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.760853 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.760857 | controller | | base64 -d)' 2026-03-20 16:26:25.760863 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.760869 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.760873 | controller | - ' exit 1' 2026-03-20 16:26:25.760877 | controller | - ' fi' 2026-03-20 16:26:25.760882 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.760886 | controller | - ' ' 2026-03-20 16:26:25.760890 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.760894 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.760899 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.760903 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.760907 | controller | --timeout=300s' 2026-03-20 16:26:25.760911 | controller | - ' ' 2026-03-20 16:26:25.760916 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.760920 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.760924 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.760929 | controller | - ' ' 2026-03-20 16:26:25.760933 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.760937 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.760949 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.760953 | controller | - ' ' 2026-03-20 16:26:25.760958 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.760962 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.760966 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.760971 | controller | available"' 2026-03-20 16:26:25.760975 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.760981 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.760986 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.760990 | controller | - ' exit 1' 2026-03-20 16:26:25.760994 | controller | - ' fi' 2026-03-20 16:26:25.760998 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.761003 | controller | - ' ' 2026-03-20 16:26:25.761007 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.761011 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.761016 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.761020 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.761024 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.761028 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.761033 | controller | -- \' 2026-03-20 16:26:25.761037 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.761041 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.761046 | controller | - ' logger.go:42: 16:13:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.761052 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.761056 | controller | - ' logger.go:42: 16:13:05 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.761060 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.761065 | controller | found' 2026-03-20 16:26:25.761069 | controller | - ' logger.go:42: 16:13:06 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.761073 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.761078 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.761082 | controller | - ' ' 2026-03-20 16:26:25.761086 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.761090 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.761094 | controller | - ' ' 2026-03-20 16:26:25.761099 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.761103 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.761107 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.761111 | controller | - ' exit 1' 2026-03-20 16:26:25.761116 | controller | - ' fi' 2026-03-20 16:26:25.761120 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.761124 | controller | - ' ' 2026-03-20 16:26:25.761129 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.761133 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.761137 | controller | | base64 -d)' 2026-03-20 16:26:25.761141 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.761146 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.761150 | controller | - ' exit 1' 2026-03-20 16:26:25.761154 | controller | - ' fi' 2026-03-20 16:26:25.761158 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.761163 | controller | - ' ' 2026-03-20 16:26:25.761167 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.761171 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.761175 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.761180 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.761184 | controller | --timeout=300s' 2026-03-20 16:26:25.761188 | controller | - ' ' 2026-03-20 16:26:25.761192 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.761196 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.761201 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.761205 | controller | - ' ' 2026-03-20 16:26:25.761209 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.761213 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.761218 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.761222 | controller | - ' ' 2026-03-20 16:26:25.761234 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.761238 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.761243 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.761247 | controller | available"' 2026-03-20 16:26:25.761251 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.761257 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.761262 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.761266 | controller | - ' exit 1' 2026-03-20 16:26:25.761270 | controller | - ' fi' 2026-03-20 16:26:25.761274 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.761278 | controller | - ' ' 2026-03-20 16:26:25.761283 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.761287 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.761291 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.761295 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.761300 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.761304 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.761308 | controller | -- \' 2026-03-20 16:26:25.761312 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.761317 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.761321 | controller | - ' logger.go:42: 16:13:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.761325 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.761329 | controller | - ' logger.go:42: 16:13:07 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.761334 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.761338 | controller | found' 2026-03-20 16:26:25.761342 | controller | - ' logger.go:42: 16:13:08 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.761346 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.761351 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.761355 | controller | - ' ' 2026-03-20 16:26:25.761359 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.761363 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.761368 | controller | - ' ' 2026-03-20 16:26:25.761372 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.761376 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.761380 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.761385 | controller | - ' exit 1' 2026-03-20 16:26:25.761389 | controller | - ' fi' 2026-03-20 16:26:25.761393 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.761397 | controller | - ' ' 2026-03-20 16:26:25.761402 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.761406 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.761410 | controller | | base64 -d)' 2026-03-20 16:26:25.761414 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.761419 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.761423 | controller | - ' exit 1' 2026-03-20 16:26:25.761429 | controller | - ' fi' 2026-03-20 16:26:25.761433 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.761438 | controller | - ' ' 2026-03-20 16:26:25.761442 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.761446 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.761452 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.761456 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.761460 | controller | --timeout=300s' 2026-03-20 16:26:25.761465 | controller | - ' ' 2026-03-20 16:26:25.761469 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.761473 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.761477 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.761481 | controller | - ' ' 2026-03-20 16:26:25.761486 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.761490 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.761494 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.761498 | controller | - ' ' 2026-03-20 16:26:25.761510 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.761514 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.761519 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.761523 | controller | available"' 2026-03-20 16:26:25.761527 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.761531 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.761536 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.761540 | controller | - ' exit 1' 2026-03-20 16:26:25.761544 | controller | - ' fi' 2026-03-20 16:26:25.761549 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.761553 | controller | - ' ' 2026-03-20 16:26:25.761559 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.761564 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.761568 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.761572 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.761576 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.761581 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.761585 | controller | -- \' 2026-03-20 16:26:25.761589 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.761593 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.761599 | controller | - ' logger.go:42: 16:13:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.761605 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.761610 | controller | - ' logger.go:42: 16:13:08 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.761616 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.761621 | controller | found' 2026-03-20 16:26:25.761625 | controller | - ' logger.go:42: 16:13:09 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.761629 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.761634 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.761638 | controller | - ' ' 2026-03-20 16:26:25.761644 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.761649 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.761653 | controller | - ' ' 2026-03-20 16:26:25.761657 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.761661 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.761666 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.761670 | controller | - ' exit 1' 2026-03-20 16:26:25.761674 | controller | - ' fi' 2026-03-20 16:26:25.761678 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.761683 | controller | - ' ' 2026-03-20 16:26:25.761687 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.761701 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.761708 | controller | | base64 -d)' 2026-03-20 16:26:25.761713 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.761717 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.761721 | controller | - ' exit 1' 2026-03-20 16:26:25.761726 | controller | - ' fi' 2026-03-20 16:26:25.761730 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.761734 | controller | - ' ' 2026-03-20 16:26:25.761739 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.761743 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.761747 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.761751 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.761756 | controller | --timeout=300s' 2026-03-20 16:26:25.761760 | controller | - ' ' 2026-03-20 16:26:25.761764 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.761768 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.761775 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.761779 | controller | - ' ' 2026-03-20 16:26:25.761784 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.761788 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.761792 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.761797 | controller | - ' ' 2026-03-20 16:26:25.761801 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.761813 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.761819 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.761824 | controller | available"' 2026-03-20 16:26:25.761828 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.761833 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.761837 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.761841 | controller | - ' exit 1' 2026-03-20 16:26:25.761845 | controller | - ' fi' 2026-03-20 16:26:25.761850 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.761854 | controller | - ' ' 2026-03-20 16:26:25.761858 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.761864 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.761869 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.761873 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.761877 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.761882 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.761886 | controller | -- \' 2026-03-20 16:26:25.761890 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.761895 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.761899 | controller | - ' logger.go:42: 16:13:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.761903 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.761908 | controller | - ' logger.go:42: 16:13:09 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.761912 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.761916 | controller | found' 2026-03-20 16:26:25.761921 | controller | - ' logger.go:42: 16:13:10 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.761925 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.761929 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.761934 | controller | - ' ' 2026-03-20 16:26:25.761941 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.761945 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.761950 | controller | - ' ' 2026-03-20 16:26:25.761954 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.761958 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.761962 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.761967 | controller | - ' exit 1' 2026-03-20 16:26:25.761971 | controller | - ' fi' 2026-03-20 16:26:25.761975 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.761979 | controller | - ' ' 2026-03-20 16:26:25.761985 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.761990 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.761994 | controller | | base64 -d)' 2026-03-20 16:26:25.761998 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.762002 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.762007 | controller | - ' exit 1' 2026-03-20 16:26:25.762011 | controller | - ' fi' 2026-03-20 16:26:25.762015 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.762019 | controller | - ' ' 2026-03-20 16:26:25.762024 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.762028 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.762032 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.762037 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.762041 | controller | --timeout=300s' 2026-03-20 16:26:25.762045 | controller | - ' ' 2026-03-20 16:26:25.762049 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.762054 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.762060 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.762064 | controller | - ' ' 2026-03-20 16:26:25.762068 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.762072 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.762077 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.762081 | controller | - ' ' 2026-03-20 16:26:25.762085 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.762090 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.762103 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.762107 | controller | available"' 2026-03-20 16:26:25.762112 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.762116 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.762120 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.762124 | controller | - ' exit 1' 2026-03-20 16:26:25.762129 | controller | - ' fi' 2026-03-20 16:26:25.762133 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.762137 | controller | - ' ' 2026-03-20 16:26:25.762142 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.762146 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.762150 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.762156 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.762161 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.762165 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.762169 | controller | -- \' 2026-03-20 16:26:25.762174 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.762178 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.762182 | controller | - ' logger.go:42: 16:13:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.762186 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.762191 | controller | - ' logger.go:42: 16:13:10 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.762195 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.762199 | controller | found' 2026-03-20 16:26:25.762203 | controller | - ' logger.go:42: 16:13:11 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.762207 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.762212 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.762216 | controller | - ' ' 2026-03-20 16:26:25.762220 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.762224 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.762229 | controller | - ' ' 2026-03-20 16:26:25.762233 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.762237 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.762241 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.762248 | controller | - ' exit 1' 2026-03-20 16:26:25.762252 | controller | - ' fi' 2026-03-20 16:26:25.762256 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.762261 | controller | - ' ' 2026-03-20 16:26:25.762265 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.762269 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.762273 | controller | | base64 -d)' 2026-03-20 16:26:25.762278 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.762282 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.762286 | controller | - ' exit 1' 2026-03-20 16:26:25.762290 | controller | - ' fi' 2026-03-20 16:26:25.762295 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.762299 | controller | - ' ' 2026-03-20 16:26:25.762303 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.762307 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.762312 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.762316 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.762320 | controller | --timeout=300s' 2026-03-20 16:26:25.762324 | controller | - ' ' 2026-03-20 16:26:25.762329 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.762333 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.762337 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.762341 | controller | - ' ' 2026-03-20 16:26:25.762346 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.762350 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.762354 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.762358 | controller | - ' ' 2026-03-20 16:26:25.762363 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.762367 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.762379 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.762383 | controller | available"' 2026-03-20 16:26:25.762387 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.762392 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.762396 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.762400 | controller | - ' exit 1' 2026-03-20 16:26:25.762404 | controller | - ' fi' 2026-03-20 16:26:25.762409 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.762413 | controller | - ' ' 2026-03-20 16:26:25.762417 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.762421 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.762426 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.762430 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.762434 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.762438 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.762444 | controller | -- \' 2026-03-20 16:26:25.762449 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.762453 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.762458 | controller | - ' logger.go:42: 16:13:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.762463 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.762469 | controller | - ' logger.go:42: 16:13:11 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.762474 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.762480 | controller | found' 2026-03-20 16:26:25.762485 | controller | - ' logger.go:42: 16:13:12 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.762491 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.762496 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.762501 | controller | - ' ' 2026-03-20 16:26:25.762507 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.762512 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.762518 | controller | - ' ' 2026-03-20 16:26:25.762523 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.762529 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.762535 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.762540 | controller | - ' exit 1' 2026-03-20 16:26:25.762545 | controller | - ' fi' 2026-03-20 16:26:25.762549 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.762553 | controller | - ' ' 2026-03-20 16:26:25.762557 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.762562 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.762566 | controller | | base64 -d)' 2026-03-20 16:26:25.762570 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.762574 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.762578 | controller | - ' exit 1' 2026-03-20 16:26:25.762582 | controller | - ' fi' 2026-03-20 16:26:25.762586 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.762591 | controller | - ' ' 2026-03-20 16:26:25.762595 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.762599 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.762603 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.762607 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.762612 | controller | --timeout=300s' 2026-03-20 16:26:25.762616 | controller | - ' ' 2026-03-20 16:26:25.762620 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.762624 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.762628 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.762632 | controller | - ' ' 2026-03-20 16:26:25.762637 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.762641 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.762645 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.762653 | controller | - ' ' 2026-03-20 16:26:25.762657 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.762662 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.762666 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.762670 | controller | available"' 2026-03-20 16:26:25.762684 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.762688 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.762705 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.762711 | controller | - ' exit 1' 2026-03-20 16:26:25.762716 | controller | - ' fi' 2026-03-20 16:26:25.762720 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.762724 | controller | - ' ' 2026-03-20 16:26:25.762728 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.762733 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.762737 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.762741 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.762748 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.762752 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.762756 | controller | -- \' 2026-03-20 16:26:25.762761 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.762766 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.762771 | controller | - ' logger.go:42: 16:13:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.762775 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.762779 | controller | - ' logger.go:42: 16:13:12 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.762783 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.762787 | controller | found' 2026-03-20 16:26:25.762792 | controller | - ' logger.go:42: 16:13:13 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.762796 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.762800 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.762804 | controller | - ' ' 2026-03-20 16:26:25.762808 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.762813 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.762817 | controller | - ' ' 2026-03-20 16:26:25.762821 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.762825 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.762830 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.762834 | controller | - ' exit 1' 2026-03-20 16:26:25.762838 | controller | - ' fi' 2026-03-20 16:26:25.762842 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.762846 | controller | - ' ' 2026-03-20 16:26:25.762851 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.762855 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.762859 | controller | | base64 -d)' 2026-03-20 16:26:25.762863 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.762870 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.762874 | controller | - ' exit 1' 2026-03-20 16:26:25.762879 | controller | - ' fi' 2026-03-20 16:26:25.762883 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.762887 | controller | - ' ' 2026-03-20 16:26:25.762891 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.762896 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.762900 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.762904 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.762908 | controller | --timeout=300s' 2026-03-20 16:26:25.762913 | controller | - ' ' 2026-03-20 16:26:25.762917 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.762921 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.762925 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.762929 | controller | - ' ' 2026-03-20 16:26:25.762934 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.762938 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.762942 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.762946 | controller | - ' ' 2026-03-20 16:26:25.762953 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.762959 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.762963 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.762967 | controller | available"' 2026-03-20 16:26:25.762971 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.762983 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.762988 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.762992 | controller | - ' exit 1' 2026-03-20 16:26:25.762997 | controller | - ' fi' 2026-03-20 16:26:25.763001 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.763005 | controller | - ' ' 2026-03-20 16:26:25.763009 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.763014 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.763018 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.763022 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.763026 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.763031 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.763035 | controller | -- \' 2026-03-20 16:26:25.763039 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.763043 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.763047 | controller | - ' logger.go:42: 16:13:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.763052 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.763059 | controller | - ' logger.go:42: 16:13:13 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.763063 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.763067 | controller | found' 2026-03-20 16:26:25.763071 | controller | - ' logger.go:42: 16:13:14 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.763076 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.763080 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.763084 | controller | - ' ' 2026-03-20 16:26:25.763088 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.763092 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.763097 | controller | - ' ' 2026-03-20 16:26:25.763101 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.763105 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.763109 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.763113 | controller | - ' exit 1' 2026-03-20 16:26:25.763118 | controller | - ' fi' 2026-03-20 16:26:25.763122 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.763126 | controller | - ' ' 2026-03-20 16:26:25.763130 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.763134 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.763139 | controller | | base64 -d)' 2026-03-20 16:26:25.763143 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.763147 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.763151 | controller | - ' exit 1' 2026-03-20 16:26:25.763156 | controller | - ' fi' 2026-03-20 16:26:25.763160 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.763176 | controller | - ' ' 2026-03-20 16:26:25.763180 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.763184 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.763188 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.763192 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.763197 | controller | --timeout=300s' 2026-03-20 16:26:25.763201 | controller | - ' ' 2026-03-20 16:26:25.763205 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.763209 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.763213 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.763218 | controller | - ' ' 2026-03-20 16:26:25.763222 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.763226 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.763230 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.763235 | controller | - ' ' 2026-03-20 16:26:25.763239 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.763243 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.763247 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.763251 | controller | available"' 2026-03-20 16:26:25.763256 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.763272 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.763277 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.763281 | controller | - ' exit 1' 2026-03-20 16:26:25.763285 | controller | - ' fi' 2026-03-20 16:26:25.763289 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.763294 | controller | - ' ' 2026-03-20 16:26:25.763298 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.763302 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.763306 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.763310 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.763315 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.763319 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.763323 | controller | -- \' 2026-03-20 16:26:25.763327 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.763332 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.763336 | controller | - ' logger.go:42: 16:13:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.763340 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.763347 | controller | - ' logger.go:42: 16:13:15 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.763351 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.763356 | controller | found' 2026-03-20 16:26:25.763360 | controller | - ' logger.go:42: 16:13:16 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.763364 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.763368 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.763372 | controller | - ' ' 2026-03-20 16:26:25.763376 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.763381 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.763385 | controller | - ' ' 2026-03-20 16:26:25.763389 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.763393 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.763397 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.763402 | controller | - ' exit 1' 2026-03-20 16:26:25.763406 | controller | - ' fi' 2026-03-20 16:26:25.763410 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.763414 | controller | - ' ' 2026-03-20 16:26:25.763418 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.763423 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.763427 | controller | | base64 -d)' 2026-03-20 16:26:25.763431 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.763435 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.763439 | controller | - ' exit 1' 2026-03-20 16:26:25.763444 | controller | - ' fi' 2026-03-20 16:26:25.763448 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.763452 | controller | - ' ' 2026-03-20 16:26:25.763456 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.763460 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.763465 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.763470 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.763475 | controller | --timeout=300s' 2026-03-20 16:26:25.763481 | controller | - ' ' 2026-03-20 16:26:25.763486 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.763491 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.763495 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.763499 | controller | - ' ' 2026-03-20 16:26:25.763503 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.763507 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.763511 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.763516 | controller | - ' ' 2026-03-20 16:26:25.763520 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.763524 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.763528 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.763532 | controller | available"' 2026-03-20 16:26:25.763537 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.763541 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.763552 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.763557 | controller | - ' exit 1' 2026-03-20 16:26:25.763561 | controller | - ' fi' 2026-03-20 16:26:25.763566 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.763570 | controller | - ' ' 2026-03-20 16:26:25.763574 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.763578 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.763582 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.763586 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.763591 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.763595 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.763599 | controller | -- \' 2026-03-20 16:26:25.763603 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.763607 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.763612 | controller | - ' logger.go:42: 16:13:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.763616 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.763620 | controller | - ' logger.go:42: 16:13:16 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.763625 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.763629 | controller | found' 2026-03-20 16:26:25.763633 | controller | - ' logger.go:42: 16:13:17 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.763637 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.763642 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.763648 | controller | - ' ' 2026-03-20 16:26:25.763652 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.763659 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.763663 | controller | - ' ' 2026-03-20 16:26:25.763687 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.763703 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.763711 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.763715 | controller | - ' exit 1' 2026-03-20 16:26:25.763719 | controller | - ' fi' 2026-03-20 16:26:25.763724 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.763728 | controller | - ' ' 2026-03-20 16:26:25.763732 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.763736 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.763740 | controller | | base64 -d)' 2026-03-20 16:26:25.763744 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.763749 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.763753 | controller | - ' exit 1' 2026-03-20 16:26:25.763757 | controller | - ' fi' 2026-03-20 16:26:25.763761 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.763765 | controller | - ' ' 2026-03-20 16:26:25.763770 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.763774 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.763778 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.763782 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.763786 | controller | --timeout=300s' 2026-03-20 16:26:25.763791 | controller | - ' ' 2026-03-20 16:26:25.763795 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.763799 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.763803 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.763807 | controller | - ' ' 2026-03-20 16:26:25.763812 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.763819 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.763823 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.763828 | controller | - ' ' 2026-03-20 16:26:25.763833 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.763837 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.763841 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.763845 | controller | available"' 2026-03-20 16:26:25.763850 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.763854 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.763858 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.763870 | controller | - ' exit 1' 2026-03-20 16:26:25.763874 | controller | - ' fi' 2026-03-20 16:26:25.763878 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.763883 | controller | - ' ' 2026-03-20 16:26:25.763887 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.763891 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.763898 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.763902 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.763906 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.763910 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.763914 | controller | -- \' 2026-03-20 16:26:25.763919 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.763923 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.763927 | controller | - ' logger.go:42: 16:13:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.763931 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.763935 | controller | - ' logger.go:42: 16:13:17 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.763941 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.763945 | controller | found' 2026-03-20 16:26:25.763950 | controller | - ' logger.go:42: 16:13:18 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.763954 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.763958 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.763962 | controller | - ' ' 2026-03-20 16:26:25.763966 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.763971 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.763975 | controller | - ' ' 2026-03-20 16:26:25.763979 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.763983 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.763987 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.763992 | controller | - ' exit 1' 2026-03-20 16:26:25.763996 | controller | - ' fi' 2026-03-20 16:26:25.764000 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.764004 | controller | - ' ' 2026-03-20 16:26:25.764008 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.764013 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.764017 | controller | | base64 -d)' 2026-03-20 16:26:25.764021 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.764025 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.764029 | controller | - ' exit 1' 2026-03-20 16:26:25.764034 | controller | - ' fi' 2026-03-20 16:26:25.764039 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.764044 | controller | - ' ' 2026-03-20 16:26:25.764048 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.764052 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.764056 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.764061 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.764065 | controller | --timeout=300s' 2026-03-20 16:26:25.764069 | controller | - ' ' 2026-03-20 16:26:25.764073 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.764077 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.764082 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.764088 | controller | - ' ' 2026-03-20 16:26:25.764092 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.764096 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.764100 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.764104 | controller | - ' ' 2026-03-20 16:26:25.764109 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.764113 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.764117 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.764121 | controller | available"' 2026-03-20 16:26:25.764125 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.764129 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.764134 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.764138 | controller | - ' exit 1' 2026-03-20 16:26:25.764142 | controller | - ' fi' 2026-03-20 16:26:25.764154 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.764159 | controller | - ' ' 2026-03-20 16:26:25.764163 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.764167 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.764171 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.764175 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.764180 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.764184 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.764188 | controller | -- \' 2026-03-20 16:26:25.764192 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.764197 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.764201 | controller | - ' logger.go:42: 16:13:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.764205 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.764209 | controller | - ' logger.go:42: 16:13:18 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.764213 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.764218 | controller | found' 2026-03-20 16:26:25.764222 | controller | - ' logger.go:42: 16:13:19 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.764226 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.764230 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.764234 | controller | - ' ' 2026-03-20 16:26:25.764238 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.764243 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.764247 | controller | - ' ' 2026-03-20 16:26:25.764251 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.764255 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.764259 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.764263 | controller | - ' exit 1' 2026-03-20 16:26:25.764268 | controller | - ' fi' 2026-03-20 16:26:25.764284 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.764289 | controller | - ' ' 2026-03-20 16:26:25.764293 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.764297 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.764302 | controller | | base64 -d)' 2026-03-20 16:26:25.764306 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.764310 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.764314 | controller | - ' exit 1' 2026-03-20 16:26:25.764318 | controller | - ' fi' 2026-03-20 16:26:25.764323 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.764327 | controller | - ' ' 2026-03-20 16:26:25.764331 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.764335 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.764340 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.764344 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.764348 | controller | --timeout=300s' 2026-03-20 16:26:25.764352 | controller | - ' ' 2026-03-20 16:26:25.764356 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.764361 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.764365 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.764369 | controller | - ' ' 2026-03-20 16:26:25.764373 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.764378 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.764382 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.764386 | controller | - ' ' 2026-03-20 16:26:25.764390 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.764394 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.764399 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.764403 | controller | available"' 2026-03-20 16:26:25.764407 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.764411 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.764415 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.764419 | controller | - ' exit 1' 2026-03-20 16:26:25.764424 | controller | - ' fi' 2026-03-20 16:26:25.764439 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.764443 | controller | - ' ' 2026-03-20 16:26:25.764448 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.764452 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.764456 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.764460 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.764464 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.764469 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.764475 | controller | -- \' 2026-03-20 16:26:25.764479 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.764483 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.764487 | controller | - ' logger.go:42: 16:13:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.764492 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.764496 | controller | - ' logger.go:42: 16:13:19 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.764500 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.764504 | controller | found' 2026-03-20 16:26:25.764510 | controller | - ' logger.go:42: 16:13:20 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.764515 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.764519 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.764523 | controller | - ' ' 2026-03-20 16:26:25.764528 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.764532 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.764536 | controller | - ' ' 2026-03-20 16:26:25.764540 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.764545 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.764549 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.764555 | controller | - ' exit 1' 2026-03-20 16:26:25.764559 | controller | - ' fi' 2026-03-20 16:26:25.764563 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.764568 | controller | - ' ' 2026-03-20 16:26:25.764572 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.764579 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.764583 | controller | | base64 -d)' 2026-03-20 16:26:25.764588 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.764592 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.764596 | controller | - ' exit 1' 2026-03-20 16:26:25.764600 | controller | - ' fi' 2026-03-20 16:26:25.764605 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.764609 | controller | - ' ' 2026-03-20 16:26:25.764613 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.764617 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.764621 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.764626 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.764630 | controller | --timeout=300s' 2026-03-20 16:26:25.764634 | controller | - ' ' 2026-03-20 16:26:25.764638 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.764642 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.764646 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.764651 | controller | - ' ' 2026-03-20 16:26:25.764655 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.764659 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.764663 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.764669 | controller | - ' ' 2026-03-20 16:26:25.764673 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.764678 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.764682 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.764686 | controller | available"' 2026-03-20 16:26:25.764690 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.764707 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.764712 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.764716 | controller | - ' exit 1' 2026-03-20 16:26:25.764721 | controller | - ' fi' 2026-03-20 16:26:25.764727 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.764732 | controller | - ' ' 2026-03-20 16:26:25.764748 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.764755 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.764760 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.764766 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.764770 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.764774 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.764781 | controller | -- \' 2026-03-20 16:26:25.764786 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.764790 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.764795 | controller | - ' logger.go:42: 16:13:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.764799 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.764803 | controller | - ' logger.go:42: 16:13:20 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.764807 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.764811 | controller | found' 2026-03-20 16:26:25.764816 | controller | - ' logger.go:42: 16:13:21 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.764820 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.764824 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.764828 | controller | - ' ' 2026-03-20 16:26:25.764833 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.764837 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.764841 | controller | - ' ' 2026-03-20 16:26:25.764845 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.764849 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.764854 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.764858 | controller | - ' exit 1' 2026-03-20 16:26:25.764862 | controller | - ' fi' 2026-03-20 16:26:25.764866 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.764870 | controller | - ' ' 2026-03-20 16:26:25.764875 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.764879 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.764883 | controller | | base64 -d)' 2026-03-20 16:26:25.764887 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.764891 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.764898 | controller | - ' exit 1' 2026-03-20 16:26:25.764902 | controller | - ' fi' 2026-03-20 16:26:25.764906 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.764910 | controller | - ' ' 2026-03-20 16:26:25.764915 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.764920 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.764926 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.764931 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.764937 | controller | --timeout=300s' 2026-03-20 16:26:25.764942 | controller | - ' ' 2026-03-20 16:26:25.764946 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.764951 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.764955 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.764959 | controller | - ' ' 2026-03-20 16:26:25.764963 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.764967 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.764972 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.764976 | controller | - ' ' 2026-03-20 16:26:25.764980 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.764984 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.764988 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.764993 | controller | available"' 2026-03-20 16:26:25.764997 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.765001 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.765005 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.765009 | controller | - ' exit 1' 2026-03-20 16:26:25.765014 | controller | - ' fi' 2026-03-20 16:26:25.765018 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.765022 | controller | - ' ' 2026-03-20 16:26:25.765034 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.765038 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.765043 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.765047 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.765051 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.765055 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.765059 | controller | -- \' 2026-03-20 16:26:25.765063 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.765068 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.765072 | controller | - ' logger.go:42: 16:13:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.765076 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.765080 | controller | - ' logger.go:42: 16:13:22 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.765087 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.765091 | controller | found' 2026-03-20 16:26:25.765095 | controller | - ' logger.go:42: 16:13:23 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.765099 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.765104 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.765108 | controller | - ' ' 2026-03-20 16:26:25.765114 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.765118 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.765123 | controller | - ' ' 2026-03-20 16:26:25.765127 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.765131 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.765135 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.765140 | controller | - ' exit 1' 2026-03-20 16:26:25.765145 | controller | - ' fi' 2026-03-20 16:26:25.765150 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.765154 | controller | - ' ' 2026-03-20 16:26:25.765158 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.765162 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.765166 | controller | | base64 -d)' 2026-03-20 16:26:25.765171 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.765175 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.765179 | controller | - ' exit 1' 2026-03-20 16:26:25.765183 | controller | - ' fi' 2026-03-20 16:26:25.765187 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.765192 | controller | - ' ' 2026-03-20 16:26:25.765196 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.765200 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.765204 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.765208 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.765212 | controller | --timeout=300s' 2026-03-20 16:26:25.765217 | controller | - ' ' 2026-03-20 16:26:25.765221 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.765225 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.765229 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.765233 | controller | - ' ' 2026-03-20 16:26:25.765238 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.765242 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.765246 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.765250 | controller | - ' ' 2026-03-20 16:26:25.765254 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.765259 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.765263 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.765267 | controller | available"' 2026-03-20 16:26:25.765271 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.765275 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.765283 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.765289 | controller | - ' exit 1' 2026-03-20 16:26:25.765294 | controller | - ' fi' 2026-03-20 16:26:25.765300 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.765305 | controller | - ' ' 2026-03-20 16:26:25.765310 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.765322 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.765327 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.765331 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.765335 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.765339 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.765345 | controller | -- \' 2026-03-20 16:26:25.765350 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.765356 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.765361 | controller | - ' logger.go:42: 16:13:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.765367 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.765371 | controller | - ' logger.go:42: 16:13:23 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.765375 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.765379 | controller | found' 2026-03-20 16:26:25.765384 | controller | - ' logger.go:42: 16:13:24 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.765388 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.765392 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.765396 | controller | - ' ' 2026-03-20 16:26:25.765400 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.765405 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.765409 | controller | - ' ' 2026-03-20 16:26:25.765413 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.765417 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.765421 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.765426 | controller | - ' exit 1' 2026-03-20 16:26:25.765431 | controller | - ' fi' 2026-03-20 16:26:25.765437 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.765443 | controller | - ' ' 2026-03-20 16:26:25.765449 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.765454 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.765460 | controller | | base64 -d)' 2026-03-20 16:26:25.765464 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.765469 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.765473 | controller | - ' exit 1' 2026-03-20 16:26:25.765478 | controller | - ' fi' 2026-03-20 16:26:25.765484 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.765489 | controller | - ' ' 2026-03-20 16:26:25.765495 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.765500 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.765506 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.765514 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.765519 | controller | --timeout=300s' 2026-03-20 16:26:25.765523 | controller | - ' ' 2026-03-20 16:26:25.765527 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.765531 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.765536 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.765540 | controller | - ' ' 2026-03-20 16:26:25.765544 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.765548 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.765552 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.765557 | controller | - ' ' 2026-03-20 16:26:25.765561 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.765565 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.765569 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.765573 | controller | available"' 2026-03-20 16:26:25.765578 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.765582 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.765586 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.765590 | controller | - ' exit 1' 2026-03-20 16:26:25.765595 | controller | - ' fi' 2026-03-20 16:26:25.765599 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.765603 | controller | - ' ' 2026-03-20 16:26:25.765607 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.765611 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.765625 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.765630 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.765634 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.765638 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.765642 | controller | -- \' 2026-03-20 16:26:25.765647 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.765651 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.765655 | controller | - ' logger.go:42: 16:13:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.765659 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.765664 | controller | - ' logger.go:42: 16:13:24 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.765668 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.765672 | controller | found' 2026-03-20 16:26:25.765676 | controller | - ' logger.go:42: 16:13:25 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.765681 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.765685 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.765704 | controller | - ' ' 2026-03-20 16:26:25.765714 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.765725 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.765730 | controller | - ' ' 2026-03-20 16:26:25.765734 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.765738 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.765742 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.765747 | controller | - ' exit 1' 2026-03-20 16:26:25.765751 | controller | - ' fi' 2026-03-20 16:26:25.765755 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.765759 | controller | - ' ' 2026-03-20 16:26:25.765764 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.765768 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.765772 | controller | | base64 -d)' 2026-03-20 16:26:25.765776 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.765780 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.765784 | controller | - ' exit 1' 2026-03-20 16:26:25.765789 | controller | - ' fi' 2026-03-20 16:26:25.765793 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.765797 | controller | - ' ' 2026-03-20 16:26:25.765801 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.765806 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.765810 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.765814 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.765818 | controller | --timeout=300s' 2026-03-20 16:26:25.765822 | controller | - ' ' 2026-03-20 16:26:25.765827 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.765831 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.765835 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.765839 | controller | - ' ' 2026-03-20 16:26:25.765843 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.765848 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.765854 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.765858 | controller | - ' ' 2026-03-20 16:26:25.765862 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.765866 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.765872 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.765876 | controller | available"' 2026-03-20 16:26:25.765880 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.765885 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.765891 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.765895 | controller | - ' exit 1' 2026-03-20 16:26:25.765899 | controller | - ' fi' 2026-03-20 16:26:25.765903 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.765909 | controller | - ' ' 2026-03-20 16:26:25.765913 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.765917 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.765933 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.765938 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.765942 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.765946 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.765951 | controller | -- \' 2026-03-20 16:26:25.765955 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.765959 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.765963 | controller | - ' logger.go:42: 16:13:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.765967 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.765972 | controller | - ' logger.go:42: 16:13:25 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.765976 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.765980 | controller | found' 2026-03-20 16:26:25.765984 | controller | - ' logger.go:42: 16:13:26 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.765989 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.765993 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.765997 | controller | - ' ' 2026-03-20 16:26:25.766001 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.766006 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.766010 | controller | - ' ' 2026-03-20 16:26:25.766014 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.766018 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.766022 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.766027 | controller | - ' exit 1' 2026-03-20 16:26:25.766031 | controller | - ' fi' 2026-03-20 16:26:25.766035 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.766039 | controller | - ' ' 2026-03-20 16:26:25.766043 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.766048 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.766052 | controller | | base64 -d)' 2026-03-20 16:26:25.766056 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.766060 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.766065 | controller | - ' exit 1' 2026-03-20 16:26:25.766069 | controller | - ' fi' 2026-03-20 16:26:25.766073 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.766077 | controller | - ' ' 2026-03-20 16:26:25.766081 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.766085 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.766090 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.766094 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.766098 | controller | --timeout=300s' 2026-03-20 16:26:25.766102 | controller | - ' ' 2026-03-20 16:26:25.766107 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.766111 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.766115 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.766123 | controller | - ' ' 2026-03-20 16:26:25.766127 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.766131 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.766135 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.766140 | controller | - ' ' 2026-03-20 16:26:25.766144 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.766148 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.766152 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.766156 | controller | available"' 2026-03-20 16:26:25.766161 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.766165 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.766169 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.766173 | controller | - ' exit 1' 2026-03-20 16:26:25.766179 | controller | - ' fi' 2026-03-20 16:26:25.766184 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.766188 | controller | - ' ' 2026-03-20 16:26:25.766192 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.766196 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.766201 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.766213 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.766219 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.766224 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.766228 | controller | -- \' 2026-03-20 16:26:25.766232 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.766236 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.766241 | controller | - ' logger.go:42: 16:13:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.766245 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.766249 | controller | - ' logger.go:42: 16:13:26 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.766253 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.766257 | controller | found' 2026-03-20 16:26:25.766262 | controller | - ' logger.go:42: 16:13:27 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.766266 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.766270 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.766274 | controller | - ' ' 2026-03-20 16:26:25.766278 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.766283 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.766287 | controller | - ' ' 2026-03-20 16:26:25.766293 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.766297 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.766301 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.766305 | controller | - ' exit 1' 2026-03-20 16:26:25.766309 | controller | - ' fi' 2026-03-20 16:26:25.766313 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.766319 | controller | - ' ' 2026-03-20 16:26:25.766324 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.766329 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.766333 | controller | | base64 -d)' 2026-03-20 16:26:25.766337 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.766342 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.766346 | controller | - ' exit 1' 2026-03-20 16:26:25.766350 | controller | - ' fi' 2026-03-20 16:26:25.766354 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.766358 | controller | - ' ' 2026-03-20 16:26:25.766363 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.766367 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.766371 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.766375 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.766380 | controller | --timeout=300s' 2026-03-20 16:26:25.766384 | controller | - ' ' 2026-03-20 16:26:25.766388 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.766392 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.766396 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.766401 | controller | - ' ' 2026-03-20 16:26:25.766405 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.766409 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.766413 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.766417 | controller | - ' ' 2026-03-20 16:26:25.766422 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.766426 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.766430 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.766434 | controller | available"' 2026-03-20 16:26:25.766438 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.766443 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.766447 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.766451 | controller | - ' exit 1' 2026-03-20 16:26:25.766455 | controller | - ' fi' 2026-03-20 16:26:25.766460 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.766464 | controller | - ' ' 2026-03-20 16:26:25.766468 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.766472 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.766476 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.766490 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.766494 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.766499 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.766503 | controller | -- \' 2026-03-20 16:26:25.766507 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.766514 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.766519 | controller | - ' logger.go:42: 16:13:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.766523 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.766527 | controller | - ' logger.go:42: 16:13:27 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.766531 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.766535 | controller | found' 2026-03-20 16:26:25.766540 | controller | - ' logger.go:42: 16:13:28 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.766544 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.766548 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.766552 | controller | - ' ' 2026-03-20 16:26:25.766557 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.766561 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.766565 | controller | - ' ' 2026-03-20 16:26:25.766569 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.766573 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.766578 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.766582 | controller | - ' exit 1' 2026-03-20 16:26:25.766586 | controller | - ' fi' 2026-03-20 16:26:25.766590 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.766595 | controller | - ' ' 2026-03-20 16:26:25.766599 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.766603 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.766607 | controller | | base64 -d)' 2026-03-20 16:26:25.766612 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.766616 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.766620 | controller | - ' exit 1' 2026-03-20 16:26:25.766624 | controller | - ' fi' 2026-03-20 16:26:25.766629 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.766633 | controller | - ' ' 2026-03-20 16:26:25.766637 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.766641 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.766645 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.766650 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.766654 | controller | --timeout=300s' 2026-03-20 16:26:25.766658 | controller | - ' ' 2026-03-20 16:26:25.766662 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.766667 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.766671 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.766675 | controller | - ' ' 2026-03-20 16:26:25.766679 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.766683 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.766688 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.766703 | controller | - ' ' 2026-03-20 16:26:25.766710 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.766716 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.766720 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.766724 | controller | available"' 2026-03-20 16:26:25.766729 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.766733 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.766737 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.766741 | controller | - ' exit 1' 2026-03-20 16:26:25.766745 | controller | - ' fi' 2026-03-20 16:26:25.766749 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.766754 | controller | - ' ' 2026-03-20 16:26:25.766758 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.766762 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.766766 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.766770 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.766784 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.766790 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.766796 | controller | -- \' 2026-03-20 16:26:25.766802 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.766807 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.766816 | controller | - ' logger.go:42: 16:13:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.766822 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.766828 | controller | - ' logger.go:42: 16:13:29 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.766833 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.766838 | controller | found' 2026-03-20 16:26:25.766844 | controller | - ' logger.go:42: 16:13:30 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.766849 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.766855 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.766867 | controller | - ' ' 2026-03-20 16:26:25.766877 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.766883 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.766889 | controller | - ' ' 2026-03-20 16:26:25.766894 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.766900 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.766906 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.766912 | controller | - ' exit 1' 2026-03-20 16:26:25.766918 | controller | - ' fi' 2026-03-20 16:26:25.766923 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.766929 | controller | - ' ' 2026-03-20 16:26:25.766934 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.766940 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.766945 | controller | | base64 -d)' 2026-03-20 16:26:25.766950 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.766956 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.766961 | controller | - ' exit 1' 2026-03-20 16:26:25.766972 | controller | - ' fi' 2026-03-20 16:26:25.766977 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.766983 | controller | - ' ' 2026-03-20 16:26:25.766988 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.766994 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.767003 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.767009 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.767015 | controller | --timeout=300s' 2026-03-20 16:26:25.767020 | controller | - ' ' 2026-03-20 16:26:25.767026 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.767032 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.767037 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.767043 | controller | - ' ' 2026-03-20 16:26:25.767049 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.767054 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.767059 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.767065 | controller | - ' ' 2026-03-20 16:26:25.767070 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.767076 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.767081 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.767087 | controller | available"' 2026-03-20 16:26:25.767092 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.767098 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.767103 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.767109 | controller | - ' exit 1' 2026-03-20 16:26:25.767114 | controller | - ' fi' 2026-03-20 16:26:25.767120 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.767125 | controller | - ' ' 2026-03-20 16:26:25.767131 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.767136 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.767140 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.767146 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.767166 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.767174 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.767180 | controller | -- \' 2026-03-20 16:26:25.767186 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.767191 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.767197 | controller | - ' logger.go:42: 16:13:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.767203 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.767209 | controller | - ' logger.go:42: 16:13:30 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.767215 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.767224 | controller | found' 2026-03-20 16:26:25.767230 | controller | - ' logger.go:42: 16:13:31 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.767236 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.767241 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.767250 | controller | - ' ' 2026-03-20 16:26:25.767255 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.767261 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.767267 | controller | - ' ' 2026-03-20 16:26:25.767272 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.767278 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.767285 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.767291 | controller | - ' exit 1' 2026-03-20 16:26:25.767297 | controller | - ' fi' 2026-03-20 16:26:25.767303 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.767309 | controller | - ' ' 2026-03-20 16:26:25.767318 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.767324 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.767330 | controller | | base64 -d)' 2026-03-20 16:26:25.767335 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.767341 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.767346 | controller | - ' exit 1' 2026-03-20 16:26:25.767352 | controller | - ' fi' 2026-03-20 16:26:25.767357 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.767363 | controller | - ' ' 2026-03-20 16:26:25.767369 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.767375 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.767381 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.767387 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.767394 | controller | --timeout=300s' 2026-03-20 16:26:25.767403 | controller | - ' ' 2026-03-20 16:26:25.767411 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.767417 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.767423 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.767429 | controller | - ' ' 2026-03-20 16:26:25.767434 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.767440 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.767446 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.767451 | controller | - ' ' 2026-03-20 16:26:25.767457 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.767463 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.767469 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.767474 | controller | available"' 2026-03-20 16:26:25.767479 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.767485 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.767491 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.767502 | controller | - ' exit 1' 2026-03-20 16:26:25.767508 | controller | - ' fi' 2026-03-20 16:26:25.767515 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.767521 | controller | - ' ' 2026-03-20 16:26:25.767527 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.767533 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.767538 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.767544 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.767550 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.767567 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.767573 | controller | -- \' 2026-03-20 16:26:25.767579 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.767585 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.767591 | controller | - ' logger.go:42: 16:13:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.767597 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.767604 | controller | - ' logger.go:42: 16:13:31 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.767610 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.767616 | controller | found' 2026-03-20 16:26:25.767621 | controller | - ' logger.go:42: 16:13:32 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.767627 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.767633 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.767639 | controller | - ' ' 2026-03-20 16:26:25.767645 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.767650 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.767656 | controller | - ' ' 2026-03-20 16:26:25.767662 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.767668 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.767707 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.767720 | controller | - ' exit 1' 2026-03-20 16:26:25.767726 | controller | - ' fi' 2026-03-20 16:26:25.767732 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.767738 | controller | - ' ' 2026-03-20 16:26:25.767744 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.767750 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.767755 | controller | | base64 -d)' 2026-03-20 16:26:25.767761 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.767767 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.767773 | controller | - ' exit 1' 2026-03-20 16:26:25.767778 | controller | - ' fi' 2026-03-20 16:26:25.767784 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.767790 | controller | - ' ' 2026-03-20 16:26:25.767796 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.767802 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.767812 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.767818 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.767828 | controller | --timeout=300s' 2026-03-20 16:26:25.767836 | controller | - ' ' 2026-03-20 16:26:25.767842 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.767848 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.767854 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.767860 | controller | - ' ' 2026-03-20 16:26:25.767865 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.767871 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.767877 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.767883 | controller | - ' ' 2026-03-20 16:26:25.767888 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.767894 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.767900 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.767905 | controller | available"' 2026-03-20 16:26:25.767911 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.767917 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.767923 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.767928 | controller | - ' exit 1' 2026-03-20 16:26:25.767934 | controller | - ' fi' 2026-03-20 16:26:25.767939 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.767945 | controller | - ' ' 2026-03-20 16:26:25.767971 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.767976 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.767980 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.767984 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.767988 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.768002 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.768007 | controller | -- \' 2026-03-20 16:26:25.768012 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.768016 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.768020 | controller | - ' logger.go:42: 16:13:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.768024 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.768028 | controller | - ' logger.go:42: 16:13:32 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.768033 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.768037 | controller | found' 2026-03-20 16:26:25.768041 | controller | - ' logger.go:42: 16:13:33 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.768045 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.768050 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.768054 | controller | - ' ' 2026-03-20 16:26:25.768058 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.768062 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.768069 | controller | - ' ' 2026-03-20 16:26:25.768073 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.768077 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.768082 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.768086 | controller | - ' exit 1' 2026-03-20 16:26:25.768090 | controller | - ' fi' 2026-03-20 16:26:25.768094 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.768099 | controller | - ' ' 2026-03-20 16:26:25.768103 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.768107 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.768111 | controller | | base64 -d)' 2026-03-20 16:26:25.768115 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.768120 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.768124 | controller | - ' exit 1' 2026-03-20 16:26:25.768128 | controller | - ' fi' 2026-03-20 16:26:25.768132 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.768136 | controller | - ' ' 2026-03-20 16:26:25.768141 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.768145 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.768149 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.768153 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.768157 | controller | --timeout=300s' 2026-03-20 16:26:25.768161 | controller | - ' ' 2026-03-20 16:26:25.768166 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.768170 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.768174 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.768178 | controller | - ' ' 2026-03-20 16:26:25.768182 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.768186 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.768191 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.768195 | controller | - ' ' 2026-03-20 16:26:25.768199 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.768206 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.768210 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.768215 | controller | available"' 2026-03-20 16:26:25.768219 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.768223 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.768227 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.768231 | controller | - ' exit 1' 2026-03-20 16:26:25.768236 | controller | - ' fi' 2026-03-20 16:26:25.768240 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.768244 | controller | - ' ' 2026-03-20 16:26:25.768248 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.768252 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.768257 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.768262 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.768267 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.768271 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.768275 | controller | -- \' 2026-03-20 16:26:25.768290 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.768295 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.768299 | controller | - ' logger.go:42: 16:13:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.768303 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.768308 | controller | - ' logger.go:42: 16:13:33 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.768312 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.768316 | controller | found' 2026-03-20 16:26:25.768320 | controller | - ' logger.go:42: 16:13:34 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.768325 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.768331 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.768336 | controller | - ' ' 2026-03-20 16:26:25.768342 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.768347 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.768353 | controller | - ' ' 2026-03-20 16:26:25.768358 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.768363 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.768369 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.768375 | controller | - ' exit 1' 2026-03-20 16:26:25.768381 | controller | - ' fi' 2026-03-20 16:26:25.768386 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.768391 | controller | - ' ' 2026-03-20 16:26:25.768397 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.768402 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.768407 | controller | | base64 -d)' 2026-03-20 16:26:25.768411 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.768415 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.768420 | controller | - ' exit 1' 2026-03-20 16:26:25.768424 | controller | - ' fi' 2026-03-20 16:26:25.768428 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.768432 | controller | - ' ' 2026-03-20 16:26:25.768436 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.768440 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.768445 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.768452 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.768456 | controller | --timeout=300s' 2026-03-20 16:26:25.768460 | controller | - ' ' 2026-03-20 16:26:25.768465 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.768469 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.768473 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.768477 | controller | - ' ' 2026-03-20 16:26:25.768484 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.768488 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.768492 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.768496 | controller | - ' ' 2026-03-20 16:26:25.768500 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.768505 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.768509 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.768513 | controller | available"' 2026-03-20 16:26:25.768517 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.768521 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.768525 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.768530 | controller | - ' exit 1' 2026-03-20 16:26:25.768534 | controller | - ' fi' 2026-03-20 16:26:25.768538 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.768542 | controller | - ' ' 2026-03-20 16:26:25.768546 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.768550 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.768555 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.768559 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.768563 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.768567 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.768571 | controller | -- \' 2026-03-20 16:26:25.768584 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.768588 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.768593 | controller | - ' logger.go:42: 16:13:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.768598 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.768602 | controller | - ' logger.go:42: 16:13:34 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.768606 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.768610 | controller | found' 2026-03-20 16:26:25.768617 | controller | - ' logger.go:42: 16:13:35 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.768622 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.768626 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.768630 | controller | - ' ' 2026-03-20 16:26:25.768634 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.768638 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.768643 | controller | - ' ' 2026-03-20 16:26:25.768647 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.768651 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.768655 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.768659 | controller | - ' exit 1' 2026-03-20 16:26:25.768663 | controller | - ' fi' 2026-03-20 16:26:25.768668 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.768672 | controller | - ' ' 2026-03-20 16:26:25.768679 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.768684 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.768688 | controller | | base64 -d)' 2026-03-20 16:26:25.768706 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.768713 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.768718 | controller | - ' exit 1' 2026-03-20 16:26:25.768722 | controller | - ' fi' 2026-03-20 16:26:25.768726 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.768730 | controller | - ' ' 2026-03-20 16:26:25.768734 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.768739 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.768743 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.768747 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.768753 | controller | --timeout=300s' 2026-03-20 16:26:25.768759 | controller | - ' ' 2026-03-20 16:26:25.768774 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.768784 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.768790 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.768795 | controller | - ' ' 2026-03-20 16:26:25.768799 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.768803 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.768807 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.768811 | controller | - ' ' 2026-03-20 16:26:25.768816 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.768820 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.768824 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.768828 | controller | available"' 2026-03-20 16:26:25.768833 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.768840 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.768845 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.768849 | controller | - ' exit 1' 2026-03-20 16:26:25.768853 | controller | - ' fi' 2026-03-20 16:26:25.768857 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.768862 | controller | - ' ' 2026-03-20 16:26:25.768866 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.768870 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.768874 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.768878 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.768882 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.768887 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.768891 | controller | -- \' 2026-03-20 16:26:25.768907 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.768915 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.768919 | controller | - ' logger.go:42: 16:13:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.768924 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.768928 | controller | - ' logger.go:42: 16:13:35 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.768932 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.768937 | controller | found' 2026-03-20 16:26:25.768941 | controller | - ' logger.go:42: 16:13:36 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.768945 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.768950 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.768954 | controller | - ' ' 2026-03-20 16:26:25.768958 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.768962 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.768969 | controller | - ' ' 2026-03-20 16:26:25.768974 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.768979 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.768983 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.768990 | controller | - ' exit 1' 2026-03-20 16:26:25.768994 | controller | - ' fi' 2026-03-20 16:26:25.768999 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.769003 | controller | - ' ' 2026-03-20 16:26:25.769007 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.769012 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.769016 | controller | | base64 -d)' 2026-03-20 16:26:25.769020 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.769025 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.769030 | controller | - ' exit 1' 2026-03-20 16:26:25.769034 | controller | - ' fi' 2026-03-20 16:26:25.769038 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.769043 | controller | - ' ' 2026-03-20 16:26:25.769047 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.769051 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.769055 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.769059 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.769064 | controller | --timeout=300s' 2026-03-20 16:26:25.769068 | controller | - ' ' 2026-03-20 16:26:25.769072 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.769076 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.769081 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.769085 | controller | - ' ' 2026-03-20 16:26:25.769089 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.769093 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.769098 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.769102 | controller | - ' ' 2026-03-20 16:26:25.769109 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.769115 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.769120 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.769124 | controller | available"' 2026-03-20 16:26:25.769128 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.769133 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.769137 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.769141 | controller | - ' exit 1' 2026-03-20 16:26:25.769145 | controller | - ' fi' 2026-03-20 16:26:25.769150 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.769154 | controller | - ' ' 2026-03-20 16:26:25.769158 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.769162 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.769167 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.769171 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.769175 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.769179 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.769184 | controller | -- \' 2026-03-20 16:26:25.769188 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.769200 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.769205 | controller | - ' logger.go:42: 16:13:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.769209 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.769213 | controller | - ' logger.go:42: 16:13:37 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.769217 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.769222 | controller | found' 2026-03-20 16:26:25.769226 | controller | - ' logger.go:42: 16:13:38 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.769230 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.769235 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.769239 | controller | - ' ' 2026-03-20 16:26:25.769243 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.769247 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.769252 | controller | - ' ' 2026-03-20 16:26:25.769256 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.769260 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.769264 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.769269 | controller | - ' exit 1' 2026-03-20 16:26:25.769273 | controller | - ' fi' 2026-03-20 16:26:25.769277 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.769281 | controller | - ' ' 2026-03-20 16:26:25.769286 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.769290 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.769294 | controller | | base64 -d)' 2026-03-20 16:26:25.769298 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.769302 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.769307 | controller | - ' exit 1' 2026-03-20 16:26:25.769311 | controller | - ' fi' 2026-03-20 16:26:25.769317 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.769321 | controller | - ' ' 2026-03-20 16:26:25.769326 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.769330 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.769334 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.769338 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.769342 | controller | --timeout=300s' 2026-03-20 16:26:25.769347 | controller | - ' ' 2026-03-20 16:26:25.769351 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.769355 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.769362 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.769367 | controller | - ' ' 2026-03-20 16:26:25.769371 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.769375 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.769380 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.769384 | controller | - ' ' 2026-03-20 16:26:25.769388 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.769392 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.769397 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.769401 | controller | available"' 2026-03-20 16:26:25.769405 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.769409 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.769413 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.769418 | controller | - ' exit 1' 2026-03-20 16:26:25.769422 | controller | - ' fi' 2026-03-20 16:26:25.769426 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.769430 | controller | - ' ' 2026-03-20 16:26:25.769435 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.769439 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.769443 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.769448 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.769452 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.769456 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.769460 | controller | -- \' 2026-03-20 16:26:25.769465 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.769469 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.769481 | controller | - ' logger.go:42: 16:13:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.769487 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.769491 | controller | - ' logger.go:42: 16:13:38 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.769495 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.769502 | controller | found' 2026-03-20 16:26:25.769507 | controller | - ' logger.go:42: 16:13:39 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.769511 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.769515 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.769520 | controller | - ' ' 2026-03-20 16:26:25.769524 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.769528 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.769533 | controller | - ' ' 2026-03-20 16:26:25.769537 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.769541 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.769545 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.769549 | controller | - ' exit 1' 2026-03-20 16:26:25.769554 | controller | - ' fi' 2026-03-20 16:26:25.769558 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.769562 | controller | - ' ' 2026-03-20 16:26:25.769566 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.769571 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.769575 | controller | | base64 -d)' 2026-03-20 16:26:25.769579 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.769583 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.769588 | controller | - ' exit 1' 2026-03-20 16:26:25.769592 | controller | - ' fi' 2026-03-20 16:26:25.769596 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.769600 | controller | - ' ' 2026-03-20 16:26:25.769605 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.769609 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.769613 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.769617 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.769622 | controller | --timeout=300s' 2026-03-20 16:26:25.769626 | controller | - ' ' 2026-03-20 16:26:25.769630 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.769634 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.769639 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.769643 | controller | - ' ' 2026-03-20 16:26:25.769647 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.769651 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.769656 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.769660 | controller | - ' ' 2026-03-20 16:26:25.769664 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.769668 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.769674 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.769679 | controller | available"' 2026-03-20 16:26:25.769683 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.769687 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.769705 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.769713 | controller | - ' exit 1' 2026-03-20 16:26:25.769733 | controller | - ' fi' 2026-03-20 16:26:25.769738 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.769744 | controller | - ' ' 2026-03-20 16:26:25.769754 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.769760 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.769767 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.769773 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.769777 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.769782 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.769786 | controller | -- \' 2026-03-20 16:26:25.769790 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.769794 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.769808 | controller | - ' logger.go:42: 16:13:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.769813 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.769818 | controller | - ' logger.go:42: 16:13:39 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.769822 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.769826 | controller | found' 2026-03-20 16:26:25.769830 | controller | - ' logger.go:42: 16:13:40 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.769834 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.769839 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.769843 | controller | - ' ' 2026-03-20 16:26:25.769847 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.769851 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.769856 | controller | - ' ' 2026-03-20 16:26:25.769860 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.769864 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.769868 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.769872 | controller | - ' exit 1' 2026-03-20 16:26:25.769877 | controller | - ' fi' 2026-03-20 16:26:25.769881 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.769885 | controller | - ' ' 2026-03-20 16:26:25.769889 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.769893 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.769897 | controller | | base64 -d)' 2026-03-20 16:26:25.769902 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.769906 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.769910 | controller | - ' exit 1' 2026-03-20 16:26:25.769914 | controller | - ' fi' 2026-03-20 16:26:25.769918 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.769923 | controller | - ' ' 2026-03-20 16:26:25.769927 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.769931 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.769935 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.769939 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.769943 | controller | --timeout=300s' 2026-03-20 16:26:25.769950 | controller | - ' ' 2026-03-20 16:26:25.769954 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.769958 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.769963 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.769967 | controller | - ' ' 2026-03-20 16:26:25.769971 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.769975 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.769979 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.769984 | controller | - ' ' 2026-03-20 16:26:25.769988 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.769992 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.769996 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.770000 | controller | available"' 2026-03-20 16:26:25.770005 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.770009 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.770013 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.770030 | controller | - ' exit 1' 2026-03-20 16:26:25.770034 | controller | - ' fi' 2026-03-20 16:26:25.770039 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.770043 | controller | - ' ' 2026-03-20 16:26:25.770049 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.770054 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.770060 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.770065 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.770070 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.770075 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.770081 | controller | -- \' 2026-03-20 16:26:25.770086 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.770092 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.770106 | controller | - ' logger.go:42: 16:13:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.770113 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.770118 | controller | - ' logger.go:42: 16:13:40 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.770124 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.770130 | controller | found' 2026-03-20 16:26:25.770135 | controller | - ' logger.go:42: 16:13:41 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.770140 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.770146 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.770152 | controller | - ' ' 2026-03-20 16:26:25.770157 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.770163 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.770169 | controller | - ' ' 2026-03-20 16:26:25.770177 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.770181 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.770185 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.770190 | controller | - ' exit 1' 2026-03-20 16:26:25.770194 | controller | - ' fi' 2026-03-20 16:26:25.770198 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.770202 | controller | - ' ' 2026-03-20 16:26:25.770212 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.770229 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.770236 | controller | | base64 -d)' 2026-03-20 16:26:25.770242 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.770248 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.770254 | controller | - ' exit 1' 2026-03-20 16:26:25.770260 | controller | - ' fi' 2026-03-20 16:26:25.770266 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.770272 | controller | - ' ' 2026-03-20 16:26:25.770289 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.770294 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.770299 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.770304 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.770309 | controller | --timeout=300s' 2026-03-20 16:26:25.770314 | controller | - ' ' 2026-03-20 16:26:25.770318 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.770323 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.770328 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.770333 | controller | - ' ' 2026-03-20 16:26:25.770337 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.770341 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.770347 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.770353 | controller | - ' ' 2026-03-20 16:26:25.770358 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.770364 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.770374 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.770378 | controller | available"' 2026-03-20 16:26:25.770382 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.770387 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.770391 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.770395 | controller | - ' exit 1' 2026-03-20 16:26:25.770399 | controller | - ' fi' 2026-03-20 16:26:25.770403 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.770407 | controller | - ' ' 2026-03-20 16:26:25.770412 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.770416 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.770421 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.770430 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.770443 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.770472 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.770479 | controller | -- \' 2026-03-20 16:26:25.770485 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.770491 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.770497 | controller | - ' logger.go:42: 16:13:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.770502 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.770514 | controller | - ' logger.go:42: 16:13:41 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.770520 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.770526 | controller | found' 2026-03-20 16:26:25.770531 | controller | - ' logger.go:42: 16:13:42 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.770536 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.770542 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.770547 | controller | - ' ' 2026-03-20 16:26:25.770552 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.770557 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.770561 | controller | - ' ' 2026-03-20 16:26:25.770565 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.770569 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.770573 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.770577 | controller | - ' exit 1' 2026-03-20 16:26:25.770582 | controller | - ' fi' 2026-03-20 16:26:25.770586 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.770590 | controller | - ' ' 2026-03-20 16:26:25.770594 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.770598 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.770603 | controller | | base64 -d)' 2026-03-20 16:26:25.770607 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.770611 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.770615 | controller | - ' exit 1' 2026-03-20 16:26:25.770619 | controller | - ' fi' 2026-03-20 16:26:25.770623 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.770628 | controller | - ' ' 2026-03-20 16:26:25.770632 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.770636 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.770640 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.770644 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.770648 | controller | --timeout=300s' 2026-03-20 16:26:25.770653 | controller | - ' ' 2026-03-20 16:26:25.770657 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.770661 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.770665 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.770669 | controller | - ' ' 2026-03-20 16:26:25.770674 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.770682 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.770687 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.770691 | controller | - ' ' 2026-03-20 16:26:25.770714 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.770718 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.770722 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.770727 | controller | available"' 2026-03-20 16:26:25.770731 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.770735 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.770739 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.770745 | controller | - ' exit 1' 2026-03-20 16:26:25.770749 | controller | - ' fi' 2026-03-20 16:26:25.770753 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.770757 | controller | - ' ' 2026-03-20 16:26:25.770762 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.770766 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.770770 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.770774 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.770778 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.770782 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.770786 | controller | -- \' 2026-03-20 16:26:25.770791 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.770795 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.770808 | controller | - ' logger.go:42: 16:13:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.770812 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.770824 | controller | - ' logger.go:42: 16:13:42 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.770829 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.770833 | controller | found' 2026-03-20 16:26:25.770837 | controller | - ' logger.go:42: 16:13:43 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.770841 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.770846 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.770850 | controller | - ' ' 2026-03-20 16:26:25.770854 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.770858 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.770862 | controller | - ' ' 2026-03-20 16:26:25.770866 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.770870 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.770875 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.770879 | controller | - ' exit 1' 2026-03-20 16:26:25.770883 | controller | - ' fi' 2026-03-20 16:26:25.770887 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.770891 | controller | - ' ' 2026-03-20 16:26:25.770896 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.770903 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.770907 | controller | | base64 -d)' 2026-03-20 16:26:25.770911 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.770915 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.770919 | controller | - ' exit 1' 2026-03-20 16:26:25.770923 | controller | - ' fi' 2026-03-20 16:26:25.770928 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.770932 | controller | - ' ' 2026-03-20 16:26:25.770936 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.770940 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.770944 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.770948 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.770953 | controller | --timeout=300s' 2026-03-20 16:26:25.770957 | controller | - ' ' 2026-03-20 16:26:25.770961 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.770965 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.770969 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.770973 | controller | - ' ' 2026-03-20 16:26:25.770981 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.770985 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.770990 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.770994 | controller | - ' ' 2026-03-20 16:26:25.770999 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.771004 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.771010 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.771016 | controller | available"' 2026-03-20 16:26:25.771022 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.771028 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.771034 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.771040 | controller | - ' exit 1' 2026-03-20 16:26:25.771046 | controller | - ' fi' 2026-03-20 16:26:25.771052 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.771057 | controller | - ' ' 2026-03-20 16:26:25.771064 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.771068 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.771073 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.771077 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.771081 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.771085 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.771089 | controller | -- \' 2026-03-20 16:26:25.771093 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.771101 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.771105 | controller | - ' logger.go:42: 16:13:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.771109 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.771113 | controller | - ' logger.go:42: 16:13:44 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.771128 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.771132 | controller | found' 2026-03-20 16:26:25.771136 | controller | - ' logger.go:42: 16:13:45 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.771141 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.771145 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.771149 | controller | - ' ' 2026-03-20 16:26:25.771153 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.771158 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.771162 | controller | - ' ' 2026-03-20 16:26:25.771166 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.771170 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.771174 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.771178 | controller | - ' exit 1' 2026-03-20 16:26:25.771183 | controller | - ' fi' 2026-03-20 16:26:25.771187 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.771191 | controller | - ' ' 2026-03-20 16:26:25.771195 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.771199 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.771203 | controller | | base64 -d)' 2026-03-20 16:26:25.771208 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.771212 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.771216 | controller | - ' exit 1' 2026-03-20 16:26:25.771220 | controller | - ' fi' 2026-03-20 16:26:25.771224 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.771228 | controller | - ' ' 2026-03-20 16:26:25.771233 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.771237 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.771241 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.771245 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.771249 | controller | --timeout=300s' 2026-03-20 16:26:25.771253 | controller | - ' ' 2026-03-20 16:26:25.771257 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.771262 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.771266 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.771270 | controller | - ' ' 2026-03-20 16:26:25.771274 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.771278 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.771282 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.771286 | controller | - ' ' 2026-03-20 16:26:25.771291 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.771295 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.771301 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.771305 | controller | available"' 2026-03-20 16:26:25.771309 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.771313 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.771317 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.771322 | controller | - ' exit 1' 2026-03-20 16:26:25.771326 | controller | - ' fi' 2026-03-20 16:26:25.771330 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.771334 | controller | - ' ' 2026-03-20 16:26:25.771339 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.771344 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.771349 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.771355 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.771360 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.771365 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.771370 | controller | -- \' 2026-03-20 16:26:25.771374 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.771378 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.771382 | controller | - ' logger.go:42: 16:13:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.771386 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.771393 | controller | - ' logger.go:42: 16:13:45 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.771398 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.771410 | controller | found' 2026-03-20 16:26:25.771415 | controller | - ' logger.go:42: 16:13:46 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.771419 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.771424 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.771428 | controller | - ' ' 2026-03-20 16:26:25.771432 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.771437 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.771441 | controller | - ' ' 2026-03-20 16:26:25.771445 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.771449 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.771453 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.771457 | controller | - ' exit 1' 2026-03-20 16:26:25.771462 | controller | - ' fi' 2026-03-20 16:26:25.771466 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.771470 | controller | - ' ' 2026-03-20 16:26:25.771474 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.771478 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.771482 | controller | | base64 -d)' 2026-03-20 16:26:25.771486 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.771491 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.771495 | controller | - ' exit 1' 2026-03-20 16:26:25.771499 | controller | - ' fi' 2026-03-20 16:26:25.771503 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.771512 | controller | - ' ' 2026-03-20 16:26:25.771516 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.771520 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.771524 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.771528 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.771533 | controller | --timeout=300s' 2026-03-20 16:26:25.771537 | controller | - ' ' 2026-03-20 16:26:25.771541 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.771545 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.771549 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.771554 | controller | - ' ' 2026-03-20 16:26:25.771558 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.771564 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.771568 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.771572 | controller | - ' ' 2026-03-20 16:26:25.771576 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.771580 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.771585 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.771590 | controller | available"' 2026-03-20 16:26:25.771594 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.771600 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.771604 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.771608 | controller | - ' exit 1' 2026-03-20 16:26:25.771612 | controller | - ' fi' 2026-03-20 16:26:25.771617 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.771621 | controller | - ' ' 2026-03-20 16:26:25.771625 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.771629 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.771633 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.771639 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.771643 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.771647 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.771651 | controller | -- \' 2026-03-20 16:26:25.771656 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.771660 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.771664 | controller | - ' logger.go:42: 16:13:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.771668 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.771672 | controller | - ' logger.go:42: 16:13:46 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.771712 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.771720 | controller | found' 2026-03-20 16:26:25.771732 | controller | - ' logger.go:42: 16:13:47 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.771739 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.771744 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.771748 | controller | - ' ' 2026-03-20 16:26:25.771753 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.771757 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.771761 | controller | - ' ' 2026-03-20 16:26:25.771765 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.771770 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.771774 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.771778 | controller | - ' exit 1' 2026-03-20 16:26:25.771782 | controller | - ' fi' 2026-03-20 16:26:25.771787 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.771791 | controller | - ' ' 2026-03-20 16:26:25.771795 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.771799 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.771803 | controller | | base64 -d)' 2026-03-20 16:26:25.771808 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.771812 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.771816 | controller | - ' exit 1' 2026-03-20 16:26:25.771820 | controller | - ' fi' 2026-03-20 16:26:25.771824 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.771831 | controller | - ' ' 2026-03-20 16:26:25.771836 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.771840 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.771845 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.771849 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.771853 | controller | --timeout=300s' 2026-03-20 16:26:25.771857 | controller | - ' ' 2026-03-20 16:26:25.771861 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.771866 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.771870 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.771874 | controller | - ' ' 2026-03-20 16:26:25.771878 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.771882 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.771887 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.771891 | controller | - ' ' 2026-03-20 16:26:25.771895 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.771899 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.771903 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.771907 | controller | available"' 2026-03-20 16:26:25.771912 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.771916 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.771920 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.771924 | controller | - ' exit 1' 2026-03-20 16:26:25.771928 | controller | - ' fi' 2026-03-20 16:26:25.771935 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.771939 | controller | - ' ' 2026-03-20 16:26:25.771943 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.771947 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.771951 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.771956 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.771960 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.771964 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.771968 | controller | -- \' 2026-03-20 16:26:25.771972 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.771977 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.771981 | controller | - ' logger.go:42: 16:13:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.771985 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.771989 | controller | - ' logger.go:42: 16:13:47 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.771993 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.771997 | controller | found' 2026-03-20 16:26:25.772002 | controller | - ' logger.go:42: 16:13:48 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.772009 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.772014 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.772018 | controller | - ' ' 2026-03-20 16:26:25.772022 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.772027 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.772031 | controller | - ' ' 2026-03-20 16:26:25.772035 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.772039 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.772044 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.772049 | controller | - ' exit 1' 2026-03-20 16:26:25.772054 | controller | - ' fi' 2026-03-20 16:26:25.772059 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.772064 | controller | - ' ' 2026-03-20 16:26:25.772068 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.772072 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.772076 | controller | | base64 -d)' 2026-03-20 16:26:25.772080 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.772085 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.772089 | controller | - ' exit 1' 2026-03-20 16:26:25.772097 | controller | - ' fi' 2026-03-20 16:26:25.772101 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.772105 | controller | - ' ' 2026-03-20 16:26:25.772110 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.772114 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.772118 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.772122 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.772126 | controller | --timeout=300s' 2026-03-20 16:26:25.772133 | controller | - ' ' 2026-03-20 16:26:25.772139 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.772143 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.772147 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.772152 | controller | - ' ' 2026-03-20 16:26:25.772156 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.772160 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.772164 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.772168 | controller | - ' ' 2026-03-20 16:26:25.772173 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.772177 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.772181 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.772185 | controller | available"' 2026-03-20 16:26:25.772189 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.772193 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.772198 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.772202 | controller | - ' exit 1' 2026-03-20 16:26:25.772206 | controller | - ' fi' 2026-03-20 16:26:25.772210 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.772214 | controller | - ' ' 2026-03-20 16:26:25.772219 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.772223 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.772229 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.772234 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.772238 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.772242 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.772246 | controller | -- \' 2026-03-20 16:26:25.772251 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.772255 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.772259 | controller | - ' logger.go:42: 16:13:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.772263 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.772269 | controller | - ' logger.go:42: 16:13:48 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.772273 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.772278 | controller | found' 2026-03-20 16:26:25.772282 | controller | - ' logger.go:42: 16:13:49 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.772286 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.772299 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.772303 | controller | - ' ' 2026-03-20 16:26:25.772307 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.772312 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.772316 | controller | - ' ' 2026-03-20 16:26:25.772321 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.772328 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.772333 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.772337 | controller | - ' exit 1' 2026-03-20 16:26:25.772341 | controller | - ' fi' 2026-03-20 16:26:25.772345 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.772349 | controller | - ' ' 2026-03-20 16:26:25.772354 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.772358 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.772362 | controller | | base64 -d)' 2026-03-20 16:26:25.772366 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.772370 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.772374 | controller | - ' exit 1' 2026-03-20 16:26:25.772378 | controller | - ' fi' 2026-03-20 16:26:25.772383 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.772387 | controller | - ' ' 2026-03-20 16:26:25.772391 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.772395 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.772399 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.772404 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.772408 | controller | --timeout=300s' 2026-03-20 16:26:25.772412 | controller | - ' ' 2026-03-20 16:26:25.772416 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.772420 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.772425 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.772429 | controller | - ' ' 2026-03-20 16:26:25.772433 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.772437 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.772442 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.772446 | controller | - ' ' 2026-03-20 16:26:25.772450 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.772454 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.772458 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.772462 | controller | available"' 2026-03-20 16:26:25.772467 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.772471 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.772475 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.772479 | controller | - ' exit 1' 2026-03-20 16:26:25.772483 | controller | - ' fi' 2026-03-20 16:26:25.772488 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.772492 | controller | - ' ' 2026-03-20 16:26:25.772496 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.772500 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.772504 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.772509 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.772515 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.772519 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.772525 | controller | -- \' 2026-03-20 16:26:25.772529 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.772533 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.772538 | controller | - ' logger.go:42: 16:13:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.772542 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.772546 | controller | - ' logger.go:42: 16:13:49 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.772550 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.772555 | controller | found' 2026-03-20 16:26:25.772559 | controller | - ' logger.go:42: 16:13:50 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.772563 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.772567 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.772571 | controller | - ' ' 2026-03-20 16:26:25.772583 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.772588 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.772592 | controller | - ' ' 2026-03-20 16:26:25.772596 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.772601 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.772605 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.772609 | controller | - ' exit 1' 2026-03-20 16:26:25.772613 | controller | - ' fi' 2026-03-20 16:26:25.772618 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.772622 | controller | - ' ' 2026-03-20 16:26:25.772626 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.772630 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.772634 | controller | | base64 -d)' 2026-03-20 16:26:25.772639 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.772643 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.772647 | controller | - ' exit 1' 2026-03-20 16:26:25.772651 | controller | - ' fi' 2026-03-20 16:26:25.772655 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.772660 | controller | - ' ' 2026-03-20 16:26:25.772664 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.772668 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.772672 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.772676 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.772681 | controller | --timeout=300s' 2026-03-20 16:26:25.772685 | controller | - ' ' 2026-03-20 16:26:25.772689 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.772710 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.772717 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.772721 | controller | - ' ' 2026-03-20 16:26:25.772725 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.772731 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.772735 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.772740 | controller | - ' ' 2026-03-20 16:26:25.772744 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.772748 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.772752 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.772756 | controller | available"' 2026-03-20 16:26:25.772761 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.772765 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.772769 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.772773 | controller | - ' exit 1' 2026-03-20 16:26:25.772778 | controller | - ' fi' 2026-03-20 16:26:25.772782 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.772786 | controller | - ' ' 2026-03-20 16:26:25.772790 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.772794 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.772799 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.772803 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.772807 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.772811 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.772815 | controller | -- \' 2026-03-20 16:26:25.772820 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.772824 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.772828 | controller | - ' logger.go:42: 16:13:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.772832 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.772839 | controller | - ' logger.go:42: 16:13:50 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.772844 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.772848 | controller | found' 2026-03-20 16:26:25.772852 | controller | - ' logger.go:42: 16:13:51 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.772856 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.772860 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.772865 | controller | - ' ' 2026-03-20 16:26:25.772876 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.772881 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.772886 | controller | - ' ' 2026-03-20 16:26:25.772890 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.772894 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.772898 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.772902 | controller | - ' exit 1' 2026-03-20 16:26:25.772907 | controller | - ' fi' 2026-03-20 16:26:25.772911 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.772915 | controller | - ' ' 2026-03-20 16:26:25.772921 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.772925 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.772932 | controller | | base64 -d)' 2026-03-20 16:26:25.772937 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.772941 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.772945 | controller | - ' exit 1' 2026-03-20 16:26:25.772949 | controller | - ' fi' 2026-03-20 16:26:25.772954 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.772958 | controller | - ' ' 2026-03-20 16:26:25.772962 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.772966 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.772970 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.772974 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.772979 | controller | --timeout=300s' 2026-03-20 16:26:25.772983 | controller | - ' ' 2026-03-20 16:26:25.772987 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.772991 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.772995 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.773000 | controller | - ' ' 2026-03-20 16:26:25.773004 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.773008 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.773012 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.773016 | controller | - ' ' 2026-03-20 16:26:25.773021 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.773025 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.773029 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.773033 | controller | available"' 2026-03-20 16:26:25.773037 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.773041 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.773045 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.773050 | controller | - ' exit 1' 2026-03-20 16:26:25.773054 | controller | - ' fi' 2026-03-20 16:26:25.773058 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.773062 | controller | - ' ' 2026-03-20 16:26:25.773068 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.773073 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.773078 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.773082 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.773086 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.773090 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.773094 | controller | -- \' 2026-03-20 16:26:25.773099 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.773103 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.773109 | controller | - ' logger.go:42: 16:13:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.773113 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.773117 | controller | - ' logger.go:42: 16:13:52 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.773121 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.773126 | controller | found' 2026-03-20 16:26:25.773130 | controller | - ' logger.go:42: 16:13:53 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.773135 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.773139 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.773143 | controller | - ' ' 2026-03-20 16:26:25.773148 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.773159 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.773165 | controller | - ' ' 2026-03-20 16:26:25.773171 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.773175 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.773180 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.773185 | controller | - ' exit 1' 2026-03-20 16:26:25.773190 | controller | - ' fi' 2026-03-20 16:26:25.773194 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.773198 | controller | - ' ' 2026-03-20 16:26:25.773202 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.773208 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.773213 | controller | | base64 -d)' 2026-03-20 16:26:25.773218 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.773222 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.773227 | controller | - ' exit 1' 2026-03-20 16:26:25.773231 | controller | - ' fi' 2026-03-20 16:26:25.773235 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.773239 | controller | - ' ' 2026-03-20 16:26:25.773243 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.773248 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.773252 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.773256 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.773260 | controller | --timeout=300s' 2026-03-20 16:26:25.773264 | controller | - ' ' 2026-03-20 16:26:25.773268 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.773273 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.773277 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.773281 | controller | - ' ' 2026-03-20 16:26:25.773285 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.773289 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.773294 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.773298 | controller | - ' ' 2026-03-20 16:26:25.773302 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.773306 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.773310 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.773317 | controller | available"' 2026-03-20 16:26:25.773321 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.773325 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.773329 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.773333 | controller | - ' exit 1' 2026-03-20 16:26:25.773337 | controller | - ' fi' 2026-03-20 16:26:25.773342 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.773347 | controller | - ' ' 2026-03-20 16:26:25.773352 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.773357 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.773361 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.773365 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.773369 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.773374 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.773378 | controller | -- \' 2026-03-20 16:26:25.773382 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.773386 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.773390 | controller | - ' logger.go:42: 16:13:53 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.773394 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.773399 | controller | - ' logger.go:42: 16:13:53 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.773406 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.773410 | controller | found' 2026-03-20 16:26:25.773414 | controller | - ' logger.go:42: 16:13:54 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.773419 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.773423 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.773427 | controller | - ' ' 2026-03-20 16:26:25.773432 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.773436 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.773447 | controller | - ' ' 2026-03-20 16:26:25.773452 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.773456 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.773460 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.773464 | controller | - ' exit 1' 2026-03-20 16:26:25.773469 | controller | - ' fi' 2026-03-20 16:26:25.773473 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.773477 | controller | - ' ' 2026-03-20 16:26:25.773481 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.773486 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.773490 | controller | | base64 -d)' 2026-03-20 16:26:25.773496 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.773502 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.773519 | controller | - ' exit 1' 2026-03-20 16:26:25.773525 | controller | - ' fi' 2026-03-20 16:26:25.773531 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.773537 | controller | - ' ' 2026-03-20 16:26:25.773547 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.773553 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.773559 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.773565 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.773571 | controller | --timeout=300s' 2026-03-20 16:26:25.773577 | controller | - ' ' 2026-03-20 16:26:25.773583 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.773588 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.773593 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.773599 | controller | - ' ' 2026-03-20 16:26:25.773604 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.773610 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.773617 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.773621 | controller | - ' ' 2026-03-20 16:26:25.773626 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.773630 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.773634 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.773638 | controller | available"' 2026-03-20 16:26:25.773643 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.773647 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.773651 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.773655 | controller | - ' exit 1' 2026-03-20 16:26:25.773659 | controller | - ' fi' 2026-03-20 16:26:25.773663 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.773668 | controller | - ' ' 2026-03-20 16:26:25.773672 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.773677 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.773690 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.773724 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.773731 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.773737 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.773742 | controller | -- \' 2026-03-20 16:26:25.773748 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.773763 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.773773 | controller | - ' logger.go:42: 16:13:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.773779 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.773783 | controller | - ' logger.go:42: 16:13:54 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.773787 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.773792 | controller | found' 2026-03-20 16:26:25.773796 | controller | - ' logger.go:42: 16:13:55 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.773803 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.773808 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.773812 | controller | - ' ' 2026-03-20 16:26:25.773816 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.773820 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.773824 | controller | - ' ' 2026-03-20 16:26:25.773841 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.773848 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.773853 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.773858 | controller | - ' exit 1' 2026-03-20 16:26:25.773862 | controller | - ' fi' 2026-03-20 16:26:25.773866 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.773870 | controller | - ' ' 2026-03-20 16:26:25.773874 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.773879 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.773883 | controller | | base64 -d)' 2026-03-20 16:26:25.773887 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.773891 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.773895 | controller | - ' exit 1' 2026-03-20 16:26:25.773900 | controller | - ' fi' 2026-03-20 16:26:25.773904 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.773908 | controller | - ' ' 2026-03-20 16:26:25.773912 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.773916 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.773921 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.773925 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.773929 | controller | --timeout=300s' 2026-03-20 16:26:25.773933 | controller | - ' ' 2026-03-20 16:26:25.773937 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.773942 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.773946 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.773950 | controller | - ' ' 2026-03-20 16:26:25.773954 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.773958 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.773962 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.773967 | controller | - ' ' 2026-03-20 16:26:25.773971 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.773978 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.773983 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.773987 | controller | available"' 2026-03-20 16:26:25.773991 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.773995 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.773999 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.774003 | controller | - ' exit 1' 2026-03-20 16:26:25.774008 | controller | - ' fi' 2026-03-20 16:26:25.774012 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.774018 | controller | - ' ' 2026-03-20 16:26:25.774022 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.774026 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.774030 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.774034 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.774039 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.774043 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.774047 | controller | -- \' 2026-03-20 16:26:25.774051 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.774055 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.774059 | controller | - ' logger.go:42: 16:13:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.774064 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.774068 | controller | - ' logger.go:42: 16:13:55 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.774072 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.774076 | controller | found' 2026-03-20 16:26:25.774084 | controller | - ' logger.go:42: 16:13:56 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.774089 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.774093 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.774097 | controller | - ' ' 2026-03-20 16:26:25.774101 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.774105 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.774110 | controller | - ' ' 2026-03-20 16:26:25.774114 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.774126 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.774132 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.774137 | controller | - ' exit 1' 2026-03-20 16:26:25.774151 | controller | - ' fi' 2026-03-20 16:26:25.774162 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.774168 | controller | - ' ' 2026-03-20 16:26:25.774174 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.774180 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.774185 | controller | | base64 -d)' 2026-03-20 16:26:25.774191 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.774197 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.774203 | controller | - ' exit 1' 2026-03-20 16:26:25.774208 | controller | - ' fi' 2026-03-20 16:26:25.774214 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.774219 | controller | - ' ' 2026-03-20 16:26:25.774224 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.774230 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.774236 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.774240 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.774244 | controller | --timeout=300s' 2026-03-20 16:26:25.774248 | controller | - ' ' 2026-03-20 16:26:25.774252 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.774260 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.774264 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.774268 | controller | - ' ' 2026-03-20 16:26:25.774272 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.774277 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.774281 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.774285 | controller | - ' ' 2026-03-20 16:26:25.774289 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.774293 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.774297 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.774303 | controller | available"' 2026-03-20 16:26:25.774313 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.774319 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.774328 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.774334 | controller | - ' exit 1' 2026-03-20 16:26:25.774340 | controller | - ' fi' 2026-03-20 16:26:25.774346 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.774352 | controller | - ' ' 2026-03-20 16:26:25.774357 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.774363 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.774367 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.774371 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.774375 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.774380 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.774384 | controller | -- \' 2026-03-20 16:26:25.774388 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.774392 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.774396 | controller | - ' logger.go:42: 16:13:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.774400 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.774405 | controller | - ' logger.go:42: 16:13:56 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.774409 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.774413 | controller | found' 2026-03-20 16:26:25.774417 | controller | - ' logger.go:42: 16:13:57 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.774421 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.774426 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.774431 | controller | - ' ' 2026-03-20 16:26:25.774445 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.774455 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.774462 | controller | - ' ' 2026-03-20 16:26:25.774468 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.774478 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.774494 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.774500 | controller | - ' exit 1' 2026-03-20 16:26:25.774504 | controller | - ' fi' 2026-03-20 16:26:25.774509 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.774513 | controller | - ' ' 2026-03-20 16:26:25.774517 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.774521 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.774525 | controller | | base64 -d)' 2026-03-20 16:26:25.774530 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.774534 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.774538 | controller | - ' exit 1' 2026-03-20 16:26:25.774544 | controller | - ' fi' 2026-03-20 16:26:25.774558 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.774568 | controller | - ' ' 2026-03-20 16:26:25.774575 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.774580 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.774586 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.774591 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.774597 | controller | --timeout=300s' 2026-03-20 16:26:25.774603 | controller | - ' ' 2026-03-20 16:26:25.774608 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.774614 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.774620 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.774626 | controller | - ' ' 2026-03-20 16:26:25.774632 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.774643 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.774656 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.774662 | controller | - ' ' 2026-03-20 16:26:25.774668 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.774674 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.774684 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.774691 | controller | available"' 2026-03-20 16:26:25.774719 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.774725 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.774729 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.774733 | controller | - ' exit 1' 2026-03-20 16:26:25.774738 | controller | - ' fi' 2026-03-20 16:26:25.774744 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.774750 | controller | - ' ' 2026-03-20 16:26:25.774764 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.774774 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.774781 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.774797 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.774804 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.774815 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.774821 | controller | -- \' 2026-03-20 16:26:25.774827 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.774833 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.774842 | controller | - ' logger.go:42: 16:13:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.774848 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.774852 | controller | - ' logger.go:42: 16:13:57 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.774856 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.774861 | controller | found' 2026-03-20 16:26:25.774865 | controller | - ' logger.go:42: 16:13:58 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.774869 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.774873 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.774877 | controller | - ' ' 2026-03-20 16:26:25.774882 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.774888 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.774892 | controller | - ' ' 2026-03-20 16:26:25.774896 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.774901 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.774916 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.774921 | controller | - ' exit 1' 2026-03-20 16:26:25.774925 | controller | - ' fi' 2026-03-20 16:26:25.774929 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.774934 | controller | - ' ' 2026-03-20 16:26:25.774938 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.774942 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.774946 | controller | | base64 -d)' 2026-03-20 16:26:25.774950 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.774954 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.774959 | controller | - ' exit 1' 2026-03-20 16:26:25.774963 | controller | - ' fi' 2026-03-20 16:26:25.774967 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.774971 | controller | - ' ' 2026-03-20 16:26:25.774975 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.774980 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.774984 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.774988 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.774992 | controller | --timeout=300s' 2026-03-20 16:26:25.774996 | controller | - ' ' 2026-03-20 16:26:25.775001 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.775005 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.775009 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.775013 | controller | - ' ' 2026-03-20 16:26:25.775017 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.775022 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.775030 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.775035 | controller | - ' ' 2026-03-20 16:26:25.775039 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.775043 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.775047 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.775052 | controller | available"' 2026-03-20 16:26:25.775056 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.775060 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.775064 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.775068 | controller | - ' exit 1' 2026-03-20 16:26:25.775073 | controller | - ' fi' 2026-03-20 16:26:25.775077 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.775081 | controller | - ' ' 2026-03-20 16:26:25.775085 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.775089 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.775094 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.775098 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.775102 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.775107 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.775111 | controller | -- \' 2026-03-20 16:26:25.775115 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.775119 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.775124 | controller | - ' logger.go:42: 16:13:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.775128 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.775132 | controller | - ' logger.go:42: 16:13:58 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.775136 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.775141 | controller | found' 2026-03-20 16:26:25.775145 | controller | - ' logger.go:42: 16:13:59 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.775149 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.775154 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.775168 | controller | - ' ' 2026-03-20 16:26:25.775173 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.775177 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.775181 | controller | - ' ' 2026-03-20 16:26:25.775186 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.775190 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.775195 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.775199 | controller | - ' exit 1' 2026-03-20 16:26:25.775203 | controller | - ' fi' 2026-03-20 16:26:25.775216 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.775221 | controller | - ' ' 2026-03-20 16:26:25.775225 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.775230 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.775237 | controller | | base64 -d)' 2026-03-20 16:26:25.775242 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.775246 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.775250 | controller | - ' exit 1' 2026-03-20 16:26:25.775255 | controller | - ' fi' 2026-03-20 16:26:25.775259 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.775263 | controller | - ' ' 2026-03-20 16:26:25.775268 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.775272 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.775276 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.775280 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.775285 | controller | --timeout=300s' 2026-03-20 16:26:25.775289 | controller | - ' ' 2026-03-20 16:26:25.775293 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.775298 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.775302 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.775306 | controller | - ' ' 2026-03-20 16:26:25.775310 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.775315 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.775319 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.775323 | controller | - ' ' 2026-03-20 16:26:25.775327 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.775332 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.775336 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.775340 | controller | available"' 2026-03-20 16:26:25.775344 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.775349 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.775353 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.775357 | controller | - ' exit 1' 2026-03-20 16:26:25.775362 | controller | - ' fi' 2026-03-20 16:26:25.775366 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.775370 | controller | - ' ' 2026-03-20 16:26:25.775375 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.775379 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.775383 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.775387 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.775392 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.775397 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.775401 | controller | -- \' 2026-03-20 16:26:25.775405 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.775410 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.775414 | controller | - ' logger.go:42: 16:13:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.775421 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.775425 | controller | - ' logger.go:42: 16:13:59 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.775429 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.775434 | controller | found' 2026-03-20 16:26:25.775438 | controller | - ' logger.go:42: 16:14:00 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.775442 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.775447 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.775451 | controller | - ' ' 2026-03-20 16:26:25.775455 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.775462 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.775467 | controller | - ' ' 2026-03-20 16:26:25.775471 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.775475 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.775480 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.775484 | controller | - ' exit 1' 2026-03-20 16:26:25.775488 | controller | - ' fi' 2026-03-20 16:26:25.775493 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.775504 | controller | - ' ' 2026-03-20 16:26:25.775509 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.775514 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.775519 | controller | | base64 -d)' 2026-03-20 16:26:25.775524 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.775533 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.775548 | controller | - ' exit 1' 2026-03-20 16:26:25.775554 | controller | - ' fi' 2026-03-20 16:26:25.775560 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.775565 | controller | - ' ' 2026-03-20 16:26:25.775575 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.775581 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.775587 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.775593 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.775598 | controller | --timeout=300s' 2026-03-20 16:26:25.775604 | controller | - ' ' 2026-03-20 16:26:25.775609 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.775615 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.775621 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.775627 | controller | - ' ' 2026-03-20 16:26:25.775632 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.775638 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.775644 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.775649 | controller | - ' ' 2026-03-20 16:26:25.775655 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.775661 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.775666 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.775676 | controller | available"' 2026-03-20 16:26:25.775732 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.775743 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.775749 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.775754 | controller | - ' exit 1' 2026-03-20 16:26:25.775759 | controller | - ' fi' 2026-03-20 16:26:25.775765 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.775771 | controller | - ' ' 2026-03-20 16:26:25.775776 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.775782 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.775787 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.775793 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.775799 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.775805 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.775811 | controller | -- \' 2026-03-20 16:26:25.775816 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.775822 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.775828 | controller | - ' logger.go:42: 16:14:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.775834 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.775839 | controller | - ' logger.go:42: 16:14:01 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.775844 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.775850 | controller | found' 2026-03-20 16:26:25.775855 | controller | - ' logger.go:42: 16:14:02 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.775861 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.775867 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.775872 | controller | - ' ' 2026-03-20 16:26:25.775878 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.775883 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.775889 | controller | - ' ' 2026-03-20 16:26:25.775894 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.775900 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.775906 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.775912 | controller | - ' exit 1' 2026-03-20 16:26:25.775918 | controller | - ' fi' 2026-03-20 16:26:25.775922 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.775926 | controller | - ' ' 2026-03-20 16:26:25.775946 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.775951 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.775955 | controller | | base64 -d)' 2026-03-20 16:26:25.775960 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.775964 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.775968 | controller | - ' exit 1' 2026-03-20 16:26:25.775972 | controller | - ' fi' 2026-03-20 16:26:25.775977 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.775981 | controller | - ' ' 2026-03-20 16:26:25.775985 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.775992 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.775996 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.776000 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.776006 | controller | --timeout=300s' 2026-03-20 16:26:25.776011 | controller | - ' ' 2026-03-20 16:26:25.776017 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.776024 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.776029 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.776035 | controller | - ' ' 2026-03-20 16:26:25.776041 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.776046 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.776052 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.776058 | controller | - ' ' 2026-03-20 16:26:25.776064 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.776072 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.776076 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.776080 | controller | available"' 2026-03-20 16:26:25.776084 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.776089 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.776093 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.776097 | controller | - ' exit 1' 2026-03-20 16:26:25.776101 | controller | - ' fi' 2026-03-20 16:26:25.776106 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.776110 | controller | - ' ' 2026-03-20 16:26:25.776114 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.776118 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.776122 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.776127 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.776131 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.776135 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.776139 | controller | -- \' 2026-03-20 16:26:25.776143 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.776148 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.776152 | controller | - ' logger.go:42: 16:14:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.776156 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.776160 | controller | - ' logger.go:42: 16:14:02 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.776164 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.776168 | controller | found' 2026-03-20 16:26:25.776173 | controller | - ' logger.go:42: 16:14:03 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.776177 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.776183 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.776188 | controller | - ' ' 2026-03-20 16:26:25.776192 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.776196 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.776200 | controller | - ' ' 2026-03-20 16:26:25.776207 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.776211 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.776215 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.776219 | controller | - ' exit 1' 2026-03-20 16:26:25.776224 | controller | - ' fi' 2026-03-20 16:26:25.776228 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.776232 | controller | - ' ' 2026-03-20 16:26:25.776236 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.776251 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.776256 | controller | | base64 -d)' 2026-03-20 16:26:25.776260 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.776264 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.776268 | controller | - ' exit 1' 2026-03-20 16:26:25.776273 | controller | - ' fi' 2026-03-20 16:26:25.776277 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.776284 | controller | - ' ' 2026-03-20 16:26:25.776288 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.776292 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.776296 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.776300 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.776305 | controller | --timeout=300s' 2026-03-20 16:26:25.776309 | controller | - ' ' 2026-03-20 16:26:25.776313 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.776317 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.776322 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.776326 | controller | - ' ' 2026-03-20 16:26:25.776330 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.776334 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.776339 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.776343 | controller | - ' ' 2026-03-20 16:26:25.776347 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.776351 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.776355 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.776360 | controller | available"' 2026-03-20 16:26:25.776364 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.776368 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.776373 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.776377 | controller | - ' exit 1' 2026-03-20 16:26:25.776381 | controller | - ' fi' 2026-03-20 16:26:25.776385 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.776389 | controller | - ' ' 2026-03-20 16:26:25.776395 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.776399 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.776404 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.776408 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.776412 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.776416 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.776420 | controller | -- \' 2026-03-20 16:26:25.776424 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.776429 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.776433 | controller | - ' logger.go:42: 16:14:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.776437 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.776442 | controller | - ' logger.go:42: 16:14:03 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.776447 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.776451 | controller | found' 2026-03-20 16:26:25.776455 | controller | - ' logger.go:42: 16:14:04 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.776459 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.776463 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.776467 | controller | - ' ' 2026-03-20 16:26:25.776472 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.776476 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.776480 | controller | - ' ' 2026-03-20 16:26:25.776484 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.776488 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.776493 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.776497 | controller | - ' exit 1' 2026-03-20 16:26:25.776501 | controller | - ' fi' 2026-03-20 16:26:25.776505 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.776509 | controller | - ' ' 2026-03-20 16:26:25.776513 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.776525 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.776530 | controller | | base64 -d)' 2026-03-20 16:26:25.776534 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.776538 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.776542 | controller | - ' exit 1' 2026-03-20 16:26:25.776546 | controller | - ' fi' 2026-03-20 16:26:25.776553 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.776558 | controller | - ' ' 2026-03-20 16:26:25.776564 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.776569 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.776574 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.776580 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.776585 | controller | --timeout=300s' 2026-03-20 16:26:25.776594 | controller | - ' ' 2026-03-20 16:26:25.776600 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.776610 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.776616 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.776622 | controller | - ' ' 2026-03-20 16:26:25.776626 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.776630 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.776634 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.776638 | controller | - ' ' 2026-03-20 16:26:25.776643 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.776647 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.776651 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.776655 | controller | available"' 2026-03-20 16:26:25.776659 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.776664 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.776668 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.776672 | controller | - ' exit 1' 2026-03-20 16:26:25.776676 | controller | - ' fi' 2026-03-20 16:26:25.776681 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.776685 | controller | - ' ' 2026-03-20 16:26:25.776689 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.776712 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.776716 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.776720 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.776725 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.776729 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.776733 | controller | -- \' 2026-03-20 16:26:25.776737 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.776742 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.776746 | controller | - ' logger.go:42: 16:14:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.776750 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.776754 | controller | - ' logger.go:42: 16:14:04 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.776758 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.776763 | controller | found' 2026-03-20 16:26:25.776767 | controller | - ' logger.go:42: 16:14:05 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.776771 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.776775 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.776779 | controller | - ' ' 2026-03-20 16:26:25.776784 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.776788 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.776792 | controller | - ' ' 2026-03-20 16:26:25.776796 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.776800 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.776807 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.776811 | controller | - ' exit 1' 2026-03-20 16:26:25.776815 | controller | - ' fi' 2026-03-20 16:26:25.776822 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.776829 | controller | - ' ' 2026-03-20 16:26:25.776833 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.776837 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.776841 | controller | | base64 -d)' 2026-03-20 16:26:25.776854 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.776859 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.776863 | controller | - ' exit 1' 2026-03-20 16:26:25.776867 | controller | - ' fi' 2026-03-20 16:26:25.776871 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.776876 | controller | - ' ' 2026-03-20 16:26:25.776880 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.776884 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.776888 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.776893 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.776897 | controller | --timeout=300s' 2026-03-20 16:26:25.776901 | controller | - ' ' 2026-03-20 16:26:25.776906 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.776912 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.776918 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.776924 | controller | - ' ' 2026-03-20 16:26:25.776930 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.776936 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.776942 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.776947 | controller | - ' ' 2026-03-20 16:26:25.776951 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.776956 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.776960 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.776964 | controller | available"' 2026-03-20 16:26:25.776968 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.776972 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.776976 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.776981 | controller | - ' exit 1' 2026-03-20 16:26:25.776985 | controller | - ' fi' 2026-03-20 16:26:25.776989 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.776993 | controller | - ' ' 2026-03-20 16:26:25.776997 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.777001 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.777006 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.777010 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.777014 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.777021 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.777026 | controller | -- \' 2026-03-20 16:26:25.777032 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.777038 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.777044 | controller | - ' logger.go:42: 16:14:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.777050 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.777056 | controller | - ' logger.go:42: 16:14:05 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.777061 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.777065 | controller | found' 2026-03-20 16:26:25.777069 | controller | - ' logger.go:42: 16:14:06 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.777073 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.777078 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.777082 | controller | - ' ' 2026-03-20 16:26:25.777086 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.777090 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.777094 | controller | - ' ' 2026-03-20 16:26:25.777099 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.777103 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.777107 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.777111 | controller | - ' exit 1' 2026-03-20 16:26:25.777115 | controller | - ' fi' 2026-03-20 16:26:25.777120 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.777124 | controller | - ' ' 2026-03-20 16:26:25.777128 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.777132 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.777136 | controller | | base64 -d)' 2026-03-20 16:26:25.777149 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.777154 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.777158 | controller | - ' exit 1' 2026-03-20 16:26:25.777162 | controller | - ' fi' 2026-03-20 16:26:25.777167 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.777171 | controller | - ' ' 2026-03-20 16:26:25.777175 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.777180 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.777184 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.777189 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.777193 | controller | --timeout=300s' 2026-03-20 16:26:25.777198 | controller | - ' ' 2026-03-20 16:26:25.777202 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.777206 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.777213 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.777218 | controller | - ' ' 2026-03-20 16:26:25.777222 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.777226 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.777234 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.777238 | controller | - ' ' 2026-03-20 16:26:25.777242 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.777246 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.777251 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.777255 | controller | available"' 2026-03-20 16:26:25.777259 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.777265 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.777269 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.777274 | controller | - ' exit 1' 2026-03-20 16:26:25.777278 | controller | - ' fi' 2026-03-20 16:26:25.777282 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.777286 | controller | - ' ' 2026-03-20 16:26:25.777290 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.777295 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.777299 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.777303 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.777307 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.777312 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.777316 | controller | -- \' 2026-03-20 16:26:25.777320 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.777325 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.777331 | controller | - ' logger.go:42: 16:14:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.777337 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.777344 | controller | - ' logger.go:42: 16:14:06 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.777350 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.777356 | controller | found' 2026-03-20 16:26:25.777361 | controller | - ' logger.go:42: 16:14:07 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.777365 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.777370 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.777374 | controller | - ' ' 2026-03-20 16:26:25.777378 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.777382 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.777390 | controller | - ' ' 2026-03-20 16:26:25.777395 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.777399 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.777403 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.777407 | controller | - ' exit 1' 2026-03-20 16:26:25.777412 | controller | - ' fi' 2026-03-20 16:26:25.777416 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.777420 | controller | - ' ' 2026-03-20 16:26:25.777425 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.777429 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.777433 | controller | | base64 -d)' 2026-03-20 16:26:25.777439 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.777453 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.777458 | controller | - ' exit 1' 2026-03-20 16:26:25.777462 | controller | - ' fi' 2026-03-20 16:26:25.777466 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.777471 | controller | - ' ' 2026-03-20 16:26:25.777475 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.777480 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.777486 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.777490 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.777495 | controller | --timeout=300s' 2026-03-20 16:26:25.777499 | controller | - ' ' 2026-03-20 16:26:25.777503 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.777508 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.777512 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.777516 | controller | - ' ' 2026-03-20 16:26:25.777520 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.777525 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.777529 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.777533 | controller | - ' ' 2026-03-20 16:26:25.777537 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.777542 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.777546 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.777550 | controller | available"' 2026-03-20 16:26:25.777554 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.777558 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.777563 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.777567 | controller | - ' exit 1' 2026-03-20 16:26:25.777571 | controller | - ' fi' 2026-03-20 16:26:25.777575 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.777580 | controller | - ' ' 2026-03-20 16:26:25.777584 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.777588 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.777593 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.777597 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.777601 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.777605 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.777610 | controller | -- \' 2026-03-20 16:26:25.777614 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.777618 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.777623 | controller | - ' logger.go:42: 16:14:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.777629 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.777633 | controller | - ' logger.go:42: 16:14:07 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.777637 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.777642 | controller | found' 2026-03-20 16:26:25.777646 | controller | - ' logger.go:42: 16:14:08 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.777650 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.777654 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.777659 | controller | - ' ' 2026-03-20 16:26:25.777663 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.777667 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.777671 | controller | - ' ' 2026-03-20 16:26:25.777676 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.777680 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.777684 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.777690 | controller | - ' exit 1' 2026-03-20 16:26:25.777712 | controller | - ' fi' 2026-03-20 16:26:25.777717 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.777721 | controller | - ' ' 2026-03-20 16:26:25.777725 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.777730 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.777734 | controller | | base64 -d)' 2026-03-20 16:26:25.777738 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.777743 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.777754 | controller | - ' exit 1' 2026-03-20 16:26:25.777759 | controller | - ' fi' 2026-03-20 16:26:25.777763 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.777767 | controller | - ' ' 2026-03-20 16:26:25.777772 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.777776 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.777780 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.777784 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.777789 | controller | --timeout=300s' 2026-03-20 16:26:25.777793 | controller | - ' ' 2026-03-20 16:26:25.777797 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.777802 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.777807 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.777811 | controller | - ' ' 2026-03-20 16:26:25.777815 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.777819 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.777824 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.777828 | controller | - ' ' 2026-03-20 16:26:25.777832 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.777836 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.777841 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.777845 | controller | available"' 2026-03-20 16:26:25.777851 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.777856 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.777860 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.777864 | controller | - ' exit 1' 2026-03-20 16:26:25.777869 | controller | - ' fi' 2026-03-20 16:26:25.777875 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.777880 | controller | - ' ' 2026-03-20 16:26:25.777884 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.777888 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.777892 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.777897 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.777901 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.777905 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.777909 | controller | -- \' 2026-03-20 16:26:25.777914 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.777918 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.777922 | controller | - ' logger.go:42: 16:14:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.777926 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.777931 | controller | - ' logger.go:42: 16:14:09 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.777935 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.777939 | controller | found' 2026-03-20 16:26:25.777943 | controller | - ' logger.go:42: 16:14:10 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.777948 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.777952 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.777956 | controller | - ' ' 2026-03-20 16:26:25.777961 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.777965 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.777969 | controller | - ' ' 2026-03-20 16:26:25.777973 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.777978 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.777982 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.777986 | controller | - ' exit 1' 2026-03-20 16:26:25.777990 | controller | - ' fi' 2026-03-20 16:26:25.777995 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.777999 | controller | - ' ' 2026-03-20 16:26:25.778003 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.778007 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.778012 | controller | | base64 -d)' 2026-03-20 16:26:25.778016 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.778020 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.778024 | controller | - ' exit 1' 2026-03-20 16:26:25.778029 | controller | - ' fi' 2026-03-20 16:26:25.778041 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.778046 | controller | - ' ' 2026-03-20 16:26:25.778050 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.778054 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.778062 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.778068 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.778073 | controller | --timeout=300s' 2026-03-20 16:26:25.778077 | controller | - ' ' 2026-03-20 16:26:25.778081 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.778085 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.778090 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.778094 | controller | - ' ' 2026-03-20 16:26:25.778098 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.778102 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.778106 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.778111 | controller | - ' ' 2026-03-20 16:26:25.778115 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.778119 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.778124 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.778128 | controller | available"' 2026-03-20 16:26:25.778132 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.778136 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.778140 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.778145 | controller | - ' exit 1' 2026-03-20 16:26:25.778149 | controller | - ' fi' 2026-03-20 16:26:25.778153 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.778157 | controller | - ' ' 2026-03-20 16:26:25.778162 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.778166 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.778170 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.778174 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.778180 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.778185 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.778189 | controller | -- \' 2026-03-20 16:26:25.778193 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.778197 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.778202 | controller | - ' logger.go:42: 16:14:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.778206 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.778210 | controller | - ' logger.go:42: 16:14:10 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.778214 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.778219 | controller | found' 2026-03-20 16:26:25.778223 | controller | - ' logger.go:42: 16:14:11 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.778227 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.778231 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.778238 | controller | - ' ' 2026-03-20 16:26:25.778243 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.778247 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.778251 | controller | - ' ' 2026-03-20 16:26:25.778255 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.778260 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.778264 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.778268 | controller | - ' exit 1' 2026-03-20 16:26:25.778272 | controller | - ' fi' 2026-03-20 16:26:25.778277 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.778281 | controller | - ' ' 2026-03-20 16:26:25.778285 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.778289 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.778294 | controller | | base64 -d)' 2026-03-20 16:26:25.778298 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.778302 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.778306 | controller | - ' exit 1' 2026-03-20 16:26:25.778311 | controller | - ' fi' 2026-03-20 16:26:25.778315 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.778327 | controller | - ' ' 2026-03-20 16:26:25.778332 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.778337 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.778341 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.778345 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.778349 | controller | --timeout=300s' 2026-03-20 16:26:25.778354 | controller | - ' ' 2026-03-20 16:26:25.778358 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.778362 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.778367 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.778371 | controller | - ' ' 2026-03-20 16:26:25.778375 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.778379 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.778384 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.778388 | controller | - ' ' 2026-03-20 16:26:25.778392 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.778396 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.778401 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.778405 | controller | available"' 2026-03-20 16:26:25.778409 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.778413 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.778418 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.778422 | controller | - ' exit 1' 2026-03-20 16:26:25.778426 | controller | - ' fi' 2026-03-20 16:26:25.778430 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.778435 | controller | - ' ' 2026-03-20 16:26:25.778439 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.778445 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.778449 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.778453 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.778458 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.778462 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.778466 | controller | -- \' 2026-03-20 16:26:25.778470 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.778475 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.778479 | controller | - ' logger.go:42: 16:14:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.778483 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.778487 | controller | - ' logger.go:42: 16:14:11 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.778492 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.778496 | controller | found' 2026-03-20 16:26:25.778500 | controller | - ' logger.go:42: 16:14:12 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.778505 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.778511 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.778517 | controller | - ' ' 2026-03-20 16:26:25.778522 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.778528 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.778534 | controller | - ' ' 2026-03-20 16:26:25.778540 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.778545 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.778551 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.778557 | controller | - ' exit 1' 2026-03-20 16:26:25.778562 | controller | - ' fi' 2026-03-20 16:26:25.778567 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.778573 | controller | - ' ' 2026-03-20 16:26:25.778603 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.778608 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.778612 | controller | | base64 -d)' 2026-03-20 16:26:25.778616 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.778621 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.778625 | controller | - ' exit 1' 2026-03-20 16:26:25.778629 | controller | - ' fi' 2026-03-20 16:26:25.778633 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.778637 | controller | - ' ' 2026-03-20 16:26:25.778650 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.778655 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.778659 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.778663 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.778667 | controller | --timeout=300s' 2026-03-20 16:26:25.778672 | controller | - ' ' 2026-03-20 16:26:25.778676 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.778680 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.778687 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.778703 | controller | - ' ' 2026-03-20 16:26:25.778710 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.778714 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.778719 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.778723 | controller | - ' ' 2026-03-20 16:26:25.778729 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.778734 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.778738 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.778742 | controller | available"' 2026-03-20 16:26:25.778746 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.778751 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.778755 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.778759 | controller | - ' exit 1' 2026-03-20 16:26:25.778763 | controller | - ' fi' 2026-03-20 16:26:25.778767 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.778772 | controller | - ' ' 2026-03-20 16:26:25.778776 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.778780 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.778784 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.778788 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.778793 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.778797 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.778801 | controller | -- \' 2026-03-20 16:26:25.778805 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.778809 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.778814 | controller | - ' logger.go:42: 16:14:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.778818 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.778822 | controller | - ' logger.go:42: 16:14:12 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.778826 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.778830 | controller | found' 2026-03-20 16:26:25.778834 | controller | - ' logger.go:42: 16:14:13 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.778839 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.778843 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.778847 | controller | - ' ' 2026-03-20 16:26:25.778851 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.778855 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.778860 | controller | - ' ' 2026-03-20 16:26:25.778864 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.778868 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.778872 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.778878 | controller | - ' exit 1' 2026-03-20 16:26:25.778883 | controller | - ' fi' 2026-03-20 16:26:25.778887 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.778891 | controller | - ' ' 2026-03-20 16:26:25.778895 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.778899 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.778904 | controller | | base64 -d)' 2026-03-20 16:26:25.778908 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.778912 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.778916 | controller | - ' exit 1' 2026-03-20 16:26:25.778921 | controller | - ' fi' 2026-03-20 16:26:25.778925 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.778929 | controller | - ' ' 2026-03-20 16:26:25.778933 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.778946 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.778951 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.778956 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.778964 | controller | --timeout=300s' 2026-03-20 16:26:25.778979 | controller | - ' ' 2026-03-20 16:26:25.778991 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.778997 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.779003 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.779009 | controller | - ' ' 2026-03-20 16:26:25.779015 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.779021 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.779027 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.779032 | controller | - ' ' 2026-03-20 16:26:25.779038 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.779043 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.779051 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.779056 | controller | available"' 2026-03-20 16:26:25.779060 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.779064 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.779068 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.779073 | controller | - ' exit 1' 2026-03-20 16:26:25.779077 | controller | - ' fi' 2026-03-20 16:26:25.779081 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.779085 | controller | - ' ' 2026-03-20 16:26:25.779089 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.779094 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.779098 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.779103 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.779109 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.779125 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.779137 | controller | -- \' 2026-03-20 16:26:25.779143 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.779149 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.779154 | controller | - ' logger.go:42: 16:14:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.779160 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.779171 | controller | - ' logger.go:42: 16:14:13 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.779176 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.779180 | controller | found' 2026-03-20 16:26:25.779184 | controller | - ' logger.go:42: 16:14:14 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.779188 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.779193 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.779197 | controller | - ' ' 2026-03-20 16:26:25.779201 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.779206 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.779210 | controller | - ' ' 2026-03-20 16:26:25.779214 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.779220 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.779225 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.779229 | controller | - ' exit 1' 2026-03-20 16:26:25.779233 | controller | - ' fi' 2026-03-20 16:26:25.779237 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.779242 | controller | - ' ' 2026-03-20 16:26:25.779246 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.779250 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.779254 | controller | | base64 -d)' 2026-03-20 16:26:25.779258 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.779262 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.779267 | controller | - ' exit 1' 2026-03-20 16:26:25.779272 | controller | - ' fi' 2026-03-20 16:26:25.779278 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.779284 | controller | - ' ' 2026-03-20 16:26:25.779290 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.779309 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.779317 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.779323 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.779329 | controller | --timeout=300s' 2026-03-20 16:26:25.779334 | controller | - ' ' 2026-03-20 16:26:25.779340 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.779344 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.779348 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.779352 | controller | - ' ' 2026-03-20 16:26:25.779357 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.779363 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.779368 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.779376 | controller | - ' ' 2026-03-20 16:26:25.779380 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.779384 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.779390 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.779394 | controller | available"' 2026-03-20 16:26:25.779398 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.779402 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.779406 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.779411 | controller | - ' exit 1' 2026-03-20 16:26:25.779415 | controller | - ' fi' 2026-03-20 16:26:25.779419 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.779423 | controller | - ' ' 2026-03-20 16:26:25.779428 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.779432 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.779436 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.779440 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.779444 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.779448 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.779453 | controller | -- \' 2026-03-20 16:26:25.779457 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.779461 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.779465 | controller | - ' logger.go:42: 16:14:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.779469 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.779474 | controller | - ' logger.go:42: 16:14:14 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.779478 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.779482 | controller | found' 2026-03-20 16:26:25.779486 | controller | - ' logger.go:42: 16:14:15 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.779490 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.779495 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.779499 | controller | - ' ' 2026-03-20 16:26:25.779503 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.779507 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.779511 | controller | - ' ' 2026-03-20 16:26:25.779515 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.779520 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.779524 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.779528 | controller | - ' exit 1' 2026-03-20 16:26:25.779532 | controller | - ' fi' 2026-03-20 16:26:25.779536 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.779541 | controller | - ' ' 2026-03-20 16:26:25.779545 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.779549 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.779553 | controller | | base64 -d)' 2026-03-20 16:26:25.779557 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.779564 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.779568 | controller | - ' exit 1' 2026-03-20 16:26:25.779573 | controller | - ' fi' 2026-03-20 16:26:25.779577 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.779581 | controller | - ' ' 2026-03-20 16:26:25.779585 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.779590 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.779602 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.779607 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.779611 | controller | --timeout=300s' 2026-03-20 16:26:25.779615 | controller | - ' ' 2026-03-20 16:26:25.779619 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.779623 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.779628 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.779632 | controller | - ' ' 2026-03-20 16:26:25.779636 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.779640 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.779644 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.779649 | controller | - ' ' 2026-03-20 16:26:25.779653 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.779657 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.779661 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.779665 | controller | available"' 2026-03-20 16:26:25.779669 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.779674 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.779678 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.779682 | controller | - ' exit 1' 2026-03-20 16:26:25.779719 | controller | - ' fi' 2026-03-20 16:26:25.779727 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.779732 | controller | - ' ' 2026-03-20 16:26:25.779736 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.779740 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.779744 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.779748 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.779753 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.779757 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.779761 | controller | -- \' 2026-03-20 16:26:25.779765 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.779769 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.779774 | controller | - ' logger.go:42: 16:14:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.779778 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.779784 | controller | - ' logger.go:42: 16:14:15 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.779788 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.779793 | controller | found' 2026-03-20 16:26:25.779797 | controller | - ' logger.go:42: 16:14:16 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.779801 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.779805 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.779809 | controller | - ' ' 2026-03-20 16:26:25.779814 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.779818 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.779822 | controller | - ' ' 2026-03-20 16:26:25.779826 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.779831 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.779835 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.779839 | controller | - ' exit 1' 2026-03-20 16:26:25.779843 | controller | - ' fi' 2026-03-20 16:26:25.779847 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.779852 | controller | - ' ' 2026-03-20 16:26:25.779856 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.779860 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.779864 | controller | | base64 -d)' 2026-03-20 16:26:25.779868 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.779873 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.779877 | controller | - ' exit 1' 2026-03-20 16:26:25.779881 | controller | - ' fi' 2026-03-20 16:26:25.779885 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.779890 | controller | - ' ' 2026-03-20 16:26:25.779894 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.779898 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.779902 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.779915 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.779920 | controller | --timeout=300s' 2026-03-20 16:26:25.779924 | controller | - ' ' 2026-03-20 16:26:25.779929 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.779933 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.779937 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.779941 | controller | - ' ' 2026-03-20 16:26:25.779946 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.779950 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.779954 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.779958 | controller | - ' ' 2026-03-20 16:26:25.779962 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.779967 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.779971 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.779975 | controller | available"' 2026-03-20 16:26:25.779979 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.779989 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.779993 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.779997 | controller | - ' exit 1' 2026-03-20 16:26:25.780001 | controller | - ' fi' 2026-03-20 16:26:25.780007 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.780011 | controller | - ' ' 2026-03-20 16:26:25.780016 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.780020 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.780024 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.780028 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.780032 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.780037 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.780041 | controller | -- \' 2026-03-20 16:26:25.780045 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.780049 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.780053 | controller | - ' logger.go:42: 16:14:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.780058 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.780062 | controller | - ' logger.go:42: 16:14:17 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.780066 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.780070 | controller | found' 2026-03-20 16:26:25.780074 | controller | - ' logger.go:42: 16:14:18 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.780078 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.780083 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.780087 | controller | - ' ' 2026-03-20 16:26:25.780091 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.780095 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.780099 | controller | - ' ' 2026-03-20 16:26:25.780104 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.780108 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.780112 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.780116 | controller | - ' exit 1' 2026-03-20 16:26:25.780121 | controller | - ' fi' 2026-03-20 16:26:25.780125 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.780129 | controller | - ' ' 2026-03-20 16:26:25.780133 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.780138 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.780142 | controller | | base64 -d)' 2026-03-20 16:26:25.780148 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.780152 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.780157 | controller | - ' exit 1' 2026-03-20 16:26:25.780161 | controller | - ' fi' 2026-03-20 16:26:25.780165 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.780169 | controller | - ' ' 2026-03-20 16:26:25.780174 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.780178 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.780182 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.780197 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.780203 | controller | --timeout=300s' 2026-03-20 16:26:25.780216 | controller | - ' ' 2026-03-20 16:26:25.780227 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.780233 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.780239 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.780244 | controller | - ' ' 2026-03-20 16:26:25.780250 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.780256 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.780262 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.780267 | controller | - ' ' 2026-03-20 16:26:25.780273 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.780279 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.780285 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.780291 | controller | available"' 2026-03-20 16:26:25.780297 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.780301 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.780305 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.780309 | controller | - ' exit 1' 2026-03-20 16:26:25.780314 | controller | - ' fi' 2026-03-20 16:26:25.780318 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.780322 | controller | - ' ' 2026-03-20 16:26:25.780326 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.780330 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.780334 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.780339 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.780343 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.780347 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.780355 | controller | -- \' 2026-03-20 16:26:25.780360 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.780364 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.780369 | controller | - ' logger.go:42: 16:14:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.780373 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.780378 | controller | - ' logger.go:42: 16:14:18 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.780382 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.780386 | controller | found' 2026-03-20 16:26:25.780390 | controller | - ' logger.go:42: 16:14:19 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.780394 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.780398 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.780403 | controller | - ' ' 2026-03-20 16:26:25.780407 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.780414 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.780418 | controller | - ' ' 2026-03-20 16:26:25.780422 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.780426 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.780430 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.780435 | controller | - ' exit 1' 2026-03-20 16:26:25.780439 | controller | - ' fi' 2026-03-20 16:26:25.780443 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.780447 | controller | - ' ' 2026-03-20 16:26:25.780451 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.780455 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.780460 | controller | | base64 -d)' 2026-03-20 16:26:25.780464 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.780468 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.780472 | controller | - ' exit 1' 2026-03-20 16:26:25.780476 | controller | - ' fi' 2026-03-20 16:26:25.780480 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.780487 | controller | - ' ' 2026-03-20 16:26:25.780492 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.780496 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.780500 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.780504 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.780509 | controller | --timeout=300s' 2026-03-20 16:26:25.780528 | controller | - ' ' 2026-03-20 16:26:25.780535 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.780540 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.780545 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.780551 | controller | - ' ' 2026-03-20 16:26:25.780557 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.780562 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.780568 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.780573 | controller | - ' ' 2026-03-20 16:26:25.780579 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.780585 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.780590 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.780596 | controller | available"' 2026-03-20 16:26:25.780603 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.780608 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.780614 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.780620 | controller | - ' exit 1' 2026-03-20 16:26:25.780626 | controller | - ' fi' 2026-03-20 16:26:25.780631 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.780637 | controller | - ' ' 2026-03-20 16:26:25.780646 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.780655 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.780661 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.780667 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.780673 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.780679 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.780685 | controller | -- \' 2026-03-20 16:26:25.780690 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.780713 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.780717 | controller | - ' logger.go:42: 16:14:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.780722 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.780726 | controller | - ' logger.go:42: 16:14:19 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.780730 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.780734 | controller | found' 2026-03-20 16:26:25.780738 | controller | - ' logger.go:42: 16:14:20 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.780743 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.780747 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.780751 | controller | - ' ' 2026-03-20 16:26:25.780755 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.780759 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.780763 | controller | - ' ' 2026-03-20 16:26:25.780768 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.780772 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.780776 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.780780 | controller | - ' exit 1' 2026-03-20 16:26:25.780784 | controller | - ' fi' 2026-03-20 16:26:25.780789 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.780793 | controller | - ' ' 2026-03-20 16:26:25.780797 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.780801 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.780805 | controller | | base64 -d)' 2026-03-20 16:26:25.780809 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.780814 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.780818 | controller | - ' exit 1' 2026-03-20 16:26:25.780822 | controller | - ' fi' 2026-03-20 16:26:25.780826 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.780830 | controller | - ' ' 2026-03-20 16:26:25.780835 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.780839 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.780843 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.780847 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.780851 | controller | --timeout=300s' 2026-03-20 16:26:25.780858 | controller | - ' ' 2026-03-20 16:26:25.780872 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.780878 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.780886 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.780890 | controller | - ' ' 2026-03-20 16:26:25.780894 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.780899 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.780903 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.780907 | controller | - ' ' 2026-03-20 16:26:25.780911 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.780916 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.780920 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.780924 | controller | available"' 2026-03-20 16:26:25.780928 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.780932 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.780937 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.780941 | controller | - ' exit 1' 2026-03-20 16:26:25.780945 | controller | - ' fi' 2026-03-20 16:26:25.780949 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.780954 | controller | - ' ' 2026-03-20 16:26:25.780958 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.780962 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.780966 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.780970 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.780975 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.780979 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.780983 | controller | -- \' 2026-03-20 16:26:25.780987 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.780991 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.780996 | controller | - ' logger.go:42: 16:14:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.781000 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.781004 | controller | - ' logger.go:42: 16:14:20 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.781008 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.781013 | controller | found' 2026-03-20 16:26:25.781017 | controller | - ' logger.go:42: 16:14:21 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.781021 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.781025 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.781029 | controller | - ' ' 2026-03-20 16:26:25.781034 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.781038 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.781042 | controller | - ' ' 2026-03-20 16:26:25.781046 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.781051 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.781055 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.781059 | controller | - ' exit 1' 2026-03-20 16:26:25.781065 | controller | - ' fi' 2026-03-20 16:26:25.781069 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.781074 | controller | - ' ' 2026-03-20 16:26:25.781078 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.781082 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.781086 | controller | | base64 -d)' 2026-03-20 16:26:25.781091 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.781095 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.781099 | controller | - ' exit 1' 2026-03-20 16:26:25.781103 | controller | - ' fi' 2026-03-20 16:26:25.781108 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.781112 | controller | - ' ' 2026-03-20 16:26:25.781116 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.781121 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.781125 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.781129 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.781133 | controller | --timeout=300s' 2026-03-20 16:26:25.781137 | controller | - ' ' 2026-03-20 16:26:25.781150 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.781154 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.781159 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.781163 | controller | - ' ' 2026-03-20 16:26:25.781167 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.781171 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.781176 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.781180 | controller | - ' ' 2026-03-20 16:26:25.781184 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.781189 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.781194 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.781200 | controller | available"' 2026-03-20 16:26:25.781206 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.781211 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.781215 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.781219 | controller | - ' exit 1' 2026-03-20 16:26:25.781224 | controller | - ' fi' 2026-03-20 16:26:25.781228 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.781232 | controller | - ' ' 2026-03-20 16:26:25.781236 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.781241 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.781245 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.781252 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.781257 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.781261 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.781268 | controller | -- \' 2026-03-20 16:26:25.781272 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.781277 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.781282 | controller | - ' logger.go:42: 16:14:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.781287 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.781292 | controller | - ' logger.go:42: 16:14:21 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.781298 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.781303 | controller | found' 2026-03-20 16:26:25.781308 | controller | - ' logger.go:42: 16:14:22 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.781312 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.781317 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.781321 | controller | - ' ' 2026-03-20 16:26:25.781329 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.781335 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.781341 | controller | - ' ' 2026-03-20 16:26:25.781347 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.781353 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.781358 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.781364 | controller | - ' exit 1' 2026-03-20 16:26:25.781370 | controller | - ' fi' 2026-03-20 16:26:25.781375 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.781381 | controller | - ' ' 2026-03-20 16:26:25.781390 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.781395 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.781399 | controller | | base64 -d)' 2026-03-20 16:26:25.781403 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.781408 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.781412 | controller | - ' exit 1' 2026-03-20 16:26:25.781416 | controller | - ' fi' 2026-03-20 16:26:25.781420 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.781425 | controller | - ' ' 2026-03-20 16:26:25.781429 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.781433 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.781437 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.781441 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.781446 | controller | --timeout=300s' 2026-03-20 16:26:25.781450 | controller | - ' ' 2026-03-20 16:26:25.781454 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.781468 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.781473 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.781477 | controller | - ' ' 2026-03-20 16:26:25.781482 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.781486 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.781504 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.781512 | controller | - ' ' 2026-03-20 16:26:25.781517 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.781523 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.781529 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.781535 | controller | available"' 2026-03-20 16:26:25.781541 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.781547 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.781551 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.781555 | controller | - ' exit 1' 2026-03-20 16:26:25.781559 | controller | - ' fi' 2026-03-20 16:26:25.781564 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.781568 | controller | - ' ' 2026-03-20 16:26:25.781572 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.781576 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.781580 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.781584 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.781589 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.781593 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.781597 | controller | -- \' 2026-03-20 16:26:25.781601 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.781605 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.781609 | controller | - ' logger.go:42: 16:14:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.781614 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.781618 | controller | - ' logger.go:42: 16:14:22 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.781622 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.781626 | controller | found' 2026-03-20 16:26:25.781630 | controller | - ' logger.go:42: 16:14:23 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.781635 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.781639 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.781643 | controller | - ' ' 2026-03-20 16:26:25.781647 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.781651 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.781656 | controller | - ' ' 2026-03-20 16:26:25.781660 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.781664 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.781668 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.781674 | controller | - ' exit 1' 2026-03-20 16:26:25.781678 | controller | - ' fi' 2026-03-20 16:26:25.781682 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.781687 | controller | - ' ' 2026-03-20 16:26:25.781691 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.781711 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.781715 | controller | | base64 -d)' 2026-03-20 16:26:25.781720 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.781724 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.781730 | controller | - ' exit 1' 2026-03-20 16:26:25.781735 | controller | - ' fi' 2026-03-20 16:26:25.781739 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.781743 | controller | - ' ' 2026-03-20 16:26:25.781748 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.781754 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.781758 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.781762 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.781767 | controller | --timeout=300s' 2026-03-20 16:26:25.781771 | controller | - ' ' 2026-03-20 16:26:25.781775 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.781789 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.781794 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.781798 | controller | - ' ' 2026-03-20 16:26:25.781802 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.781806 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.781811 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.781815 | controller | - ' ' 2026-03-20 16:26:25.781819 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.781824 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.781828 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.781832 | controller | available"' 2026-03-20 16:26:25.781837 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.781841 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.781845 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.781849 | controller | - ' exit 1' 2026-03-20 16:26:25.781854 | controller | - ' fi' 2026-03-20 16:26:25.781858 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.781862 | controller | - ' ' 2026-03-20 16:26:25.781866 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.781871 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.781875 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.781879 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.781883 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.781888 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.781892 | controller | -- \' 2026-03-20 16:26:25.781896 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.781903 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.781908 | controller | - ' logger.go:42: 16:14:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.781912 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.781917 | controller | - ' logger.go:42: 16:14:23 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.781923 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.781927 | controller | found' 2026-03-20 16:26:25.781932 | controller | - ' logger.go:42: 16:14:24 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.781936 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.781940 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.781944 | controller | - ' ' 2026-03-20 16:26:25.781949 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.781953 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.781957 | controller | - ' ' 2026-03-20 16:26:25.781962 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.781966 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.781970 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.781976 | controller | - ' exit 1' 2026-03-20 16:26:25.781980 | controller | - ' fi' 2026-03-20 16:26:25.781985 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.781989 | controller | - ' ' 2026-03-20 16:26:25.781993 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.781998 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.782002 | controller | | base64 -d)' 2026-03-20 16:26:25.782006 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.782010 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.782015 | controller | - ' exit 1' 2026-03-20 16:26:25.782019 | controller | - ' fi' 2026-03-20 16:26:25.782023 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.782028 | controller | - ' ' 2026-03-20 16:26:25.782032 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.782036 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.782040 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.782045 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.782049 | controller | --timeout=300s' 2026-03-20 16:26:25.782053 | controller | - ' ' 2026-03-20 16:26:25.782058 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.782062 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.782074 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.782079 | controller | - ' ' 2026-03-20 16:26:25.782084 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.782088 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.782092 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.782096 | controller | - ' ' 2026-03-20 16:26:25.782101 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.782105 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.782109 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.782114 | controller | available"' 2026-03-20 16:26:25.782118 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.782122 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.782128 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.782133 | controller | - ' exit 1' 2026-03-20 16:26:25.782137 | controller | - ' fi' 2026-03-20 16:26:25.782141 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.782146 | controller | - ' ' 2026-03-20 16:26:25.782150 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.782154 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.782159 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.782163 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.782167 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.782171 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.782176 | controller | -- \' 2026-03-20 16:26:25.782180 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.782184 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.782189 | controller | - ' logger.go:42: 16:14:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.782193 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.782197 | controller | - ' logger.go:42: 16:14:25 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.782201 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.782206 | controller | found' 2026-03-20 16:26:25.782210 | controller | - ' logger.go:42: 16:14:26 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.782214 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.782219 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.782223 | controller | - ' ' 2026-03-20 16:26:25.782227 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.782232 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.782236 | controller | - ' ' 2026-03-20 16:26:25.782240 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.782244 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.782249 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.782253 | controller | - ' exit 1' 2026-03-20 16:26:25.782257 | controller | - ' fi' 2026-03-20 16:26:25.782262 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.782266 | controller | - ' ' 2026-03-20 16:26:25.782270 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.782274 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.782278 | controller | | base64 -d)' 2026-03-20 16:26:25.782283 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.782287 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.782291 | controller | - ' exit 1' 2026-03-20 16:26:25.782296 | controller | - ' fi' 2026-03-20 16:26:25.782301 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.782306 | controller | - ' ' 2026-03-20 16:26:25.782311 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.782315 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.782321 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.782328 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.782332 | controller | --timeout=300s' 2026-03-20 16:26:25.782336 | controller | - ' ' 2026-03-20 16:26:25.782341 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.782345 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.782357 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.782362 | controller | - ' ' 2026-03-20 16:26:25.782367 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.782371 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.782375 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.782380 | controller | - ' ' 2026-03-20 16:26:25.782388 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.782393 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.782397 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.782401 | controller | available"' 2026-03-20 16:26:25.782406 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.782410 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.782414 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.782419 | controller | - ' exit 1' 2026-03-20 16:26:25.782423 | controller | - ' fi' 2026-03-20 16:26:25.782427 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.782431 | controller | - ' ' 2026-03-20 16:26:25.782436 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.782440 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.782444 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.782448 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.782453 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.782457 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.782461 | controller | -- \' 2026-03-20 16:26:25.782465 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.782470 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.782476 | controller | - ' logger.go:42: 16:14:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.782480 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.782485 | controller | - ' logger.go:42: 16:14:26 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.782490 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.782495 | controller | found' 2026-03-20 16:26:25.782499 | controller | - ' logger.go:42: 16:14:27 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.782503 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.782507 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.782512 | controller | - ' ' 2026-03-20 16:26:25.782516 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.782523 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.782527 | controller | - ' ' 2026-03-20 16:26:25.782532 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.782536 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.782540 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.782545 | controller | - ' exit 1' 2026-03-20 16:26:25.782549 | controller | - ' fi' 2026-03-20 16:26:25.782553 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.782557 | controller | - ' ' 2026-03-20 16:26:25.782562 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.782566 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.782570 | controller | | base64 -d)' 2026-03-20 16:26:25.782575 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.782579 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.782583 | controller | - ' exit 1' 2026-03-20 16:26:25.782587 | controller | - ' fi' 2026-03-20 16:26:25.782592 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.782596 | controller | - ' ' 2026-03-20 16:26:25.782600 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.782604 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.782609 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.782613 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.782617 | controller | --timeout=300s' 2026-03-20 16:26:25.782621 | controller | - ' ' 2026-03-20 16:26:25.782626 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.782630 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.782634 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.782646 | controller | - ' ' 2026-03-20 16:26:25.782651 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.782656 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.782660 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.782664 | controller | - ' ' 2026-03-20 16:26:25.782668 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.782673 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.782677 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.782681 | controller | available"' 2026-03-20 16:26:25.782685 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.782703 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.782711 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.782715 | controller | - ' exit 1' 2026-03-20 16:26:25.782719 | controller | - ' fi' 2026-03-20 16:26:25.782723 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.782728 | controller | - ' ' 2026-03-20 16:26:25.782732 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.782736 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.782749 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.782754 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.782758 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.782762 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.782766 | controller | -- \' 2026-03-20 16:26:25.782771 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.782775 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.782779 | controller | - ' logger.go:42: 16:14:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.782783 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.782788 | controller | - ' logger.go:42: 16:14:27 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.782792 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.782796 | controller | found' 2026-03-20 16:26:25.782800 | controller | - ' logger.go:42: 16:14:28 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.782804 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.782808 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.782815 | controller | - ' ' 2026-03-20 16:26:25.782819 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.782823 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.782828 | controller | - ' ' 2026-03-20 16:26:25.782832 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.782836 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.782841 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.782845 | controller | - ' exit 1' 2026-03-20 16:26:25.782849 | controller | - ' fi' 2026-03-20 16:26:25.782853 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.782857 | controller | - ' ' 2026-03-20 16:26:25.782862 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.782866 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.782870 | controller | | base64 -d)' 2026-03-20 16:26:25.782875 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.782879 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.782883 | controller | - ' exit 1' 2026-03-20 16:26:25.782887 | controller | - ' fi' 2026-03-20 16:26:25.782891 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.782896 | controller | - ' ' 2026-03-20 16:26:25.782900 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.782904 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.782908 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.782913 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.782917 | controller | --timeout=300s' 2026-03-20 16:26:25.782921 | controller | - ' ' 2026-03-20 16:26:25.782925 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.782930 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.782934 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.782940 | controller | - ' ' 2026-03-20 16:26:25.782953 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.782957 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.782961 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.782966 | controller | - ' ' 2026-03-20 16:26:25.782970 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.782975 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.782979 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.782983 | controller | available"' 2026-03-20 16:26:25.782987 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.782991 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.782996 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.783000 | controller | - ' exit 1' 2026-03-20 16:26:25.783004 | controller | - ' fi' 2026-03-20 16:26:25.783008 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.783013 | controller | - ' ' 2026-03-20 16:26:25.783017 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.783021 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.783025 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.783030 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.783034 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.783038 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.783042 | controller | -- \' 2026-03-20 16:26:25.783047 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.783051 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.783055 | controller | - ' logger.go:42: 16:14:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.783059 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.783064 | controller | - ' logger.go:42: 16:14:28 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.783070 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.783074 | controller | found' 2026-03-20 16:26:25.783078 | controller | - ' logger.go:42: 16:14:29 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.783082 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.783087 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.783091 | controller | - ' ' 2026-03-20 16:26:25.783095 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.783099 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.783103 | controller | - ' ' 2026-03-20 16:26:25.783108 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.783112 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.783116 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.783120 | controller | - ' exit 1' 2026-03-20 16:26:25.783124 | controller | - ' fi' 2026-03-20 16:26:25.783129 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.783135 | controller | - ' ' 2026-03-20 16:26:25.783139 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.783143 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.783148 | controller | | base64 -d)' 2026-03-20 16:26:25.783152 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.783156 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.783160 | controller | - ' exit 1' 2026-03-20 16:26:25.783164 | controller | - ' fi' 2026-03-20 16:26:25.783169 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.783173 | controller | - ' ' 2026-03-20 16:26:25.783177 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.783181 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.783186 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.783190 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.783194 | controller | --timeout=300s' 2026-03-20 16:26:25.783198 | controller | - ' ' 2026-03-20 16:26:25.783202 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.783206 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.783211 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.783215 | controller | - ' ' 2026-03-20 16:26:25.783229 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.783234 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.783238 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.783242 | controller | - ' ' 2026-03-20 16:26:25.783247 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.783251 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.783255 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.783259 | controller | available"' 2026-03-20 16:26:25.783264 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.783268 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.783272 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.783276 | controller | - ' exit 1' 2026-03-20 16:26:25.783281 | controller | - ' fi' 2026-03-20 16:26:25.783285 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.783289 | controller | - ' ' 2026-03-20 16:26:25.783293 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.783297 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.783302 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.783306 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.783310 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.783314 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.783319 | controller | -- \' 2026-03-20 16:26:25.783325 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.783329 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.783333 | controller | - ' logger.go:42: 16:14:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.783337 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.783341 | controller | - ' logger.go:42: 16:14:29 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.783346 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.783350 | controller | found' 2026-03-20 16:26:25.783354 | controller | - ' logger.go:42: 16:14:30 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.783358 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.783362 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.783367 | controller | - ' ' 2026-03-20 16:26:25.783371 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.783375 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.783379 | controller | - ' ' 2026-03-20 16:26:25.783384 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.783388 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.783392 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.783396 | controller | - ' exit 1' 2026-03-20 16:26:25.783400 | controller | - ' fi' 2026-03-20 16:26:25.783405 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.783409 | controller | - ' ' 2026-03-20 16:26:25.783413 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.783417 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.783421 | controller | | base64 -d)' 2026-03-20 16:26:25.783426 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.783430 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.783434 | controller | - ' exit 1' 2026-03-20 16:26:25.783438 | controller | - ' fi' 2026-03-20 16:26:25.783445 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.783449 | controller | - ' ' 2026-03-20 16:26:25.783454 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.783458 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.783462 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.783466 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.783470 | controller | --timeout=300s' 2026-03-20 16:26:25.783474 | controller | - ' ' 2026-03-20 16:26:25.783479 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.783483 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.783487 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.783491 | controller | - ' ' 2026-03-20 16:26:25.783495 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.783507 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.783512 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.783516 | controller | - ' ' 2026-03-20 16:26:25.783523 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.783530 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.783535 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.783539 | controller | available"' 2026-03-20 16:26:25.783543 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.783547 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.783552 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.783556 | controller | - ' exit 1' 2026-03-20 16:26:25.783560 | controller | - ' fi' 2026-03-20 16:26:25.783564 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.783569 | controller | - ' ' 2026-03-20 16:26:25.783573 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.783577 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.783581 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.783585 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.783590 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.783595 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.783601 | controller | -- \' 2026-03-20 16:26:25.783606 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.783612 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.783618 | controller | - ' logger.go:42: 16:14:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.783624 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.783630 | controller | - ' logger.go:42: 16:14:30 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.783636 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.783642 | controller | found' 2026-03-20 16:26:25.783653 | controller | - ' logger.go:42: 16:14:31 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.783659 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.783665 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.783671 | controller | - ' ' 2026-03-20 16:26:25.783675 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.783679 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.783683 | controller | - ' ' 2026-03-20 16:26:25.783688 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.783734 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.783740 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.783745 | controller | - ' exit 1' 2026-03-20 16:26:25.783750 | controller | - ' fi' 2026-03-20 16:26:25.783755 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.783760 | controller | - ' ' 2026-03-20 16:26:25.783765 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.783770 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.783775 | controller | | base64 -d)' 2026-03-20 16:26:25.783780 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.783785 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.783793 | controller | - ' exit 1' 2026-03-20 16:26:25.783798 | controller | - ' fi' 2026-03-20 16:26:25.783803 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.783808 | controller | - ' ' 2026-03-20 16:26:25.783813 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.783818 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.783823 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.783828 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.783833 | controller | --timeout=300s' 2026-03-20 16:26:25.783838 | controller | - ' ' 2026-03-20 16:26:25.783843 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.783848 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.783853 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.783858 | controller | - ' ' 2026-03-20 16:26:25.783863 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.783878 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.783884 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.783889 | controller | - ' ' 2026-03-20 16:26:25.783894 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.783899 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.783904 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.783909 | controller | available"' 2026-03-20 16:26:25.783914 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.783919 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.783924 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.783928 | controller | - ' exit 1' 2026-03-20 16:26:25.783933 | controller | - ' fi' 2026-03-20 16:26:25.783938 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.783943 | controller | - ' ' 2026-03-20 16:26:25.783949 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.783953 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.783957 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.783962 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.783966 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.783970 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.783975 | controller | -- \' 2026-03-20 16:26:25.783979 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.783983 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.783987 | controller | - ' logger.go:42: 16:14:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.783991 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.783996 | controller | - ' logger.go:42: 16:14:32 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.784002 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.784006 | controller | found' 2026-03-20 16:26:25.784011 | controller | - ' logger.go:42: 16:14:33 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.784015 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.784019 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.784023 | controller | - ' ' 2026-03-20 16:26:25.784028 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.784032 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.784036 | controller | - ' ' 2026-03-20 16:26:25.784040 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.784045 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.784049 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.784053 | controller | - ' exit 1' 2026-03-20 16:26:25.784058 | controller | - ' fi' 2026-03-20 16:26:25.784062 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.784066 | controller | - ' ' 2026-03-20 16:26:25.784073 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.784077 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.784081 | controller | | base64 -d)' 2026-03-20 16:26:25.784086 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.784090 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.784094 | controller | - ' exit 1' 2026-03-20 16:26:25.784099 | controller | - ' fi' 2026-03-20 16:26:25.784103 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.784107 | controller | - ' ' 2026-03-20 16:26:25.784111 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.784115 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.784120 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.784124 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.784128 | controller | --timeout=300s' 2026-03-20 16:26:25.784132 | controller | - ' ' 2026-03-20 16:26:25.784137 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.784141 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.784145 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.784149 | controller | - ' ' 2026-03-20 16:26:25.784154 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.784158 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.784170 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.784174 | controller | - ' ' 2026-03-20 16:26:25.784179 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.784183 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.784187 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.784193 | controller | available"' 2026-03-20 16:26:25.784198 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.784202 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.784206 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.784213 | controller | - ' exit 1' 2026-03-20 16:26:25.784218 | controller | - ' fi' 2026-03-20 16:26:25.784222 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.784226 | controller | - ' ' 2026-03-20 16:26:25.784230 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.784235 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.784239 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.784243 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.784247 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.784252 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.784256 | controller | -- \' 2026-03-20 16:26:25.784260 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.784265 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.784269 | controller | - ' logger.go:42: 16:14:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.784273 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.784277 | controller | - ' logger.go:42: 16:14:33 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.784281 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.784286 | controller | found' 2026-03-20 16:26:25.784290 | controller | - ' logger.go:42: 16:14:34 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.784294 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.784298 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.784303 | controller | - ' ' 2026-03-20 16:26:25.784309 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.784313 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.784317 | controller | - ' ' 2026-03-20 16:26:25.784321 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.784326 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.784330 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.784334 | controller | - ' exit 1' 2026-03-20 16:26:25.784338 | controller | - ' fi' 2026-03-20 16:26:25.784343 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.784347 | controller | - ' ' 2026-03-20 16:26:25.784351 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.784355 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.784360 | controller | | base64 -d)' 2026-03-20 16:26:25.784364 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.784368 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.784372 | controller | - ' exit 1' 2026-03-20 16:26:25.784377 | controller | - ' fi' 2026-03-20 16:26:25.784381 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.784385 | controller | - ' ' 2026-03-20 16:26:25.784389 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.784394 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.784398 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.784402 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.784408 | controller | --timeout=300s' 2026-03-20 16:26:25.784412 | controller | - ' ' 2026-03-20 16:26:25.784416 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.784421 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.784425 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.784429 | controller | - ' ' 2026-03-20 16:26:25.784433 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.784438 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.784453 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.784460 | controller | - ' ' 2026-03-20 16:26:25.784465 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.784471 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.784477 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.784483 | controller | available"' 2026-03-20 16:26:25.784489 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.784494 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.784500 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.784504 | controller | - ' exit 1' 2026-03-20 16:26:25.784508 | controller | - ' fi' 2026-03-20 16:26:25.784513 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.784517 | controller | - ' ' 2026-03-20 16:26:25.784521 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.784525 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.784529 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.784533 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.784538 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.784542 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.784546 | controller | -- \' 2026-03-20 16:26:25.784550 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.784554 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.784559 | controller | - ' logger.go:42: 16:14:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.784563 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.784567 | controller | - ' logger.go:42: 16:14:34 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.784571 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.784575 | controller | found' 2026-03-20 16:26:25.784579 | controller | - ' logger.go:42: 16:14:35 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.784584 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.784588 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.784592 | controller | - ' ' 2026-03-20 16:26:25.784596 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.784600 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.784607 | controller | - ' ' 2026-03-20 16:26:25.784611 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.784615 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.784619 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.784624 | controller | - ' exit 1' 2026-03-20 16:26:25.784630 | controller | - ' fi' 2026-03-20 16:26:25.784636 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.784640 | controller | - ' ' 2026-03-20 16:26:25.784644 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.784648 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.784652 | controller | | base64 -d)' 2026-03-20 16:26:25.784657 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.784661 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.784667 | controller | - ' exit 1' 2026-03-20 16:26:25.784673 | controller | - ' fi' 2026-03-20 16:26:25.784678 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.784687 | controller | - ' ' 2026-03-20 16:26:25.784706 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.784713 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.784718 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.784724 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.784729 | controller | --timeout=300s' 2026-03-20 16:26:25.784735 | controller | - ' ' 2026-03-20 16:26:25.784740 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.784746 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.784754 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.784760 | controller | - ' ' 2026-03-20 16:26:25.784765 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.784771 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.784777 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.784782 | controller | - ' ' 2026-03-20 16:26:25.784799 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.784805 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.784811 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.784817 | controller | available"' 2026-03-20 16:26:25.784822 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.784828 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.784833 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.784838 | controller | - ' exit 1' 2026-03-20 16:26:25.784844 | controller | - ' fi' 2026-03-20 16:26:25.784849 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.784854 | controller | - ' ' 2026-03-20 16:26:25.784860 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.784865 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.784871 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.784880 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.784886 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.784892 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.784898 | controller | -- \' 2026-03-20 16:26:25.784904 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.784910 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.784916 | controller | - ' logger.go:42: 16:14:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.784922 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.784928 | controller | - ' logger.go:42: 16:14:35 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.784932 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.784936 | controller | found' 2026-03-20 16:26:25.784940 | controller | - ' logger.go:42: 16:14:36 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.784945 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.784949 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.784953 | controller | - ' ' 2026-03-20 16:26:25.784957 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.784964 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.784968 | controller | - ' ' 2026-03-20 16:26:25.784973 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.784977 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.784981 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.784985 | controller | - ' exit 1' 2026-03-20 16:26:25.784989 | controller | - ' fi' 2026-03-20 16:26:25.784994 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.784998 | controller | - ' ' 2026-03-20 16:26:25.785002 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.785006 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.785010 | controller | | base64 -d)' 2026-03-20 16:26:25.785015 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.785019 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.785023 | controller | - ' exit 1' 2026-03-20 16:26:25.785027 | controller | - ' fi' 2026-03-20 16:26:25.785031 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.785035 | controller | - ' ' 2026-03-20 16:26:25.785040 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.785044 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.785048 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.785052 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.785056 | controller | --timeout=300s' 2026-03-20 16:26:25.785062 | controller | - ' ' 2026-03-20 16:26:25.785067 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.785072 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.785076 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.785084 | controller | - ' ' 2026-03-20 16:26:25.785088 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.785092 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.785096 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.785100 | controller | - ' ' 2026-03-20 16:26:25.785114 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.785120 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.785124 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.785128 | controller | available"' 2026-03-20 16:26:25.785133 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.785137 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.785141 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.785145 | controller | - ' exit 1' 2026-03-20 16:26:25.785149 | controller | - ' fi' 2026-03-20 16:26:25.785154 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.785158 | controller | - ' ' 2026-03-20 16:26:25.785162 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.785166 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.785170 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.785175 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.785179 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.785183 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.785187 | controller | -- \' 2026-03-20 16:26:25.785191 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.785195 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.785199 | controller | - ' logger.go:42: 16:14:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.785204 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.785208 | controller | - ' logger.go:42: 16:14:36 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.785212 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.785216 | controller | found' 2026-03-20 16:26:25.785220 | controller | - ' logger.go:42: 16:14:37 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.785224 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.785229 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.785233 | controller | - ' ' 2026-03-20 16:26:25.785237 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.785241 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.785245 | controller | - ' ' 2026-03-20 16:26:25.785250 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.785254 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.785258 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.785262 | controller | - ' exit 1' 2026-03-20 16:26:25.785266 | controller | - ' fi' 2026-03-20 16:26:25.785271 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.785277 | controller | - ' ' 2026-03-20 16:26:25.785281 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.785285 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.785289 | controller | | base64 -d)' 2026-03-20 16:26:25.785293 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.785297 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.785302 | controller | - ' exit 1' 2026-03-20 16:26:25.785306 | controller | - ' fi' 2026-03-20 16:26:25.785310 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.785314 | controller | - ' ' 2026-03-20 16:26:25.785318 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.785322 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.785327 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.785331 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.785335 | controller | --timeout=300s' 2026-03-20 16:26:25.785339 | controller | - ' ' 2026-03-20 16:26:25.785343 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.785347 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.785351 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.785356 | controller | - ' ' 2026-03-20 16:26:25.785360 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.785364 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.785368 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.785372 | controller | - ' ' 2026-03-20 16:26:25.785376 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.785388 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.785394 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.785398 | controller | available"' 2026-03-20 16:26:25.785402 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.785409 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.785415 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.785421 | controller | - ' exit 1' 2026-03-20 16:26:25.785426 | controller | - ' fi' 2026-03-20 16:26:25.785458 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.785465 | controller | - ' ' 2026-03-20 16:26:25.785471 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.785477 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.785483 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.785488 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.785494 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.785499 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.785505 | controller | -- \' 2026-03-20 16:26:25.785511 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.785521 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.785528 | controller | - ' logger.go:42: 16:14:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.785534 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.785540 | controller | - ' logger.go:42: 16:14:37 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.785545 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.785551 | controller | found' 2026-03-20 16:26:25.785557 | controller | - ' logger.go:42: 16:14:38 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.785563 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.785567 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.785572 | controller | - ' ' 2026-03-20 16:26:25.785576 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.785580 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.785584 | controller | - ' ' 2026-03-20 16:26:25.785592 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.785596 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.785600 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.785604 | controller | - ' exit 1' 2026-03-20 16:26:25.785609 | controller | - ' fi' 2026-03-20 16:26:25.785613 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.785617 | controller | - ' ' 2026-03-20 16:26:25.785621 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.785625 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.785629 | controller | | base64 -d)' 2026-03-20 16:26:25.785634 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.785638 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.785642 | controller | - ' exit 1' 2026-03-20 16:26:25.785646 | controller | - ' fi' 2026-03-20 16:26:25.785650 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.785655 | controller | - ' ' 2026-03-20 16:26:25.785659 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.785663 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.785667 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.785671 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.785675 | controller | --timeout=300s' 2026-03-20 16:26:25.785680 | controller | - ' ' 2026-03-20 16:26:25.785708 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.785727 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.785732 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.785736 | controller | - ' ' 2026-03-20 16:26:25.785741 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.785745 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.785751 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.785756 | controller | - ' ' 2026-03-20 16:26:25.785762 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.785772 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.785791 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.785798 | controller | available"' 2026-03-20 16:26:25.785803 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.785808 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.785812 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.785816 | controller | - ' exit 1' 2026-03-20 16:26:25.785824 | controller | - ' fi' 2026-03-20 16:26:25.785828 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.785832 | controller | - ' ' 2026-03-20 16:26:25.785836 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.785841 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.785845 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.785849 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.785855 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.785859 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.785863 | controller | -- \' 2026-03-20 16:26:25.785868 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.785872 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.785876 | controller | - ' logger.go:42: 16:14:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.785880 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.785884 | controller | - ' logger.go:42: 16:14:38 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.785888 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.785893 | controller | found' 2026-03-20 16:26:25.785897 | controller | - ' logger.go:42: 16:14:40 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.785901 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.785905 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.785909 | controller | - ' ' 2026-03-20 16:26:25.785914 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.785918 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.785922 | controller | - ' ' 2026-03-20 16:26:25.785926 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.785930 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.785934 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.785939 | controller | - ' exit 1' 2026-03-20 16:26:25.785943 | controller | - ' fi' 2026-03-20 16:26:25.785947 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.785951 | controller | - ' ' 2026-03-20 16:26:25.785955 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.785960 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.785964 | controller | | base64 -d)' 2026-03-20 16:26:25.785968 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.785972 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.785976 | controller | - ' exit 1' 2026-03-20 16:26:25.785980 | controller | - ' fi' 2026-03-20 16:26:25.785988 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.785992 | controller | - ' ' 2026-03-20 16:26:25.785996 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.786017 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.786021 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.786025 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.786030 | controller | --timeout=300s' 2026-03-20 16:26:25.786035 | controller | - ' ' 2026-03-20 16:26:25.786039 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.786043 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.786047 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.786051 | controller | - ' ' 2026-03-20 16:26:25.786056 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.786060 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.786064 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.786068 | controller | - ' ' 2026-03-20 16:26:25.786072 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.786076 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.786089 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.786094 | controller | available"' 2026-03-20 16:26:25.786099 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.786103 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.786107 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.786111 | controller | - ' exit 1' 2026-03-20 16:26:25.786115 | controller | - ' fi' 2026-03-20 16:26:25.786120 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.786124 | controller | - ' ' 2026-03-20 16:26:25.786128 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.786132 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.786136 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.786140 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.786145 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.786149 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.786153 | controller | -- \' 2026-03-20 16:26:25.786157 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.786161 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.786165 | controller | - ' logger.go:42: 16:14:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.786170 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.786174 | controller | - ' logger.go:42: 16:14:40 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.786178 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.786184 | controller | found' 2026-03-20 16:26:25.786188 | controller | - ' logger.go:42: 16:14:41 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.786192 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.786196 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.786201 | controller | - ' ' 2026-03-20 16:26:25.786205 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.786209 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.786213 | controller | - ' ' 2026-03-20 16:26:25.786217 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.786221 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.786226 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.786233 | controller | - ' exit 1' 2026-03-20 16:26:25.786237 | controller | - ' fi' 2026-03-20 16:26:25.786241 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.786245 | controller | - ' ' 2026-03-20 16:26:25.786250 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.786254 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.786258 | controller | | base64 -d)' 2026-03-20 16:26:25.786262 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.786268 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.786272 | controller | - ' exit 1' 2026-03-20 16:26:25.786276 | controller | - ' fi' 2026-03-20 16:26:25.786281 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.786285 | controller | - ' ' 2026-03-20 16:26:25.786289 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.786293 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.786297 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.786301 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.786306 | controller | --timeout=300s' 2026-03-20 16:26:25.786310 | controller | - ' ' 2026-03-20 16:26:25.786314 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.786318 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.786322 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.786326 | controller | - ' ' 2026-03-20 16:26:25.786331 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.786335 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.786339 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.786343 | controller | - ' ' 2026-03-20 16:26:25.786347 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.786351 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.786356 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.786360 | controller | available"' 2026-03-20 16:26:25.786371 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.786376 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.786380 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.786386 | controller | - ' exit 1' 2026-03-20 16:26:25.786390 | controller | - ' fi' 2026-03-20 16:26:25.786395 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.786399 | controller | - ' ' 2026-03-20 16:26:25.786403 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.786407 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.786411 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.786415 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.786419 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.786424 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.786428 | controller | -- \' 2026-03-20 16:26:25.786433 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.786437 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.786441 | controller | - ' logger.go:42: 16:14:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.786445 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.786450 | controller | - ' logger.go:42: 16:14:41 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.786454 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.786458 | controller | found' 2026-03-20 16:26:25.786462 | controller | - ' logger.go:42: 16:14:42 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.786466 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.786470 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.786475 | controller | - ' ' 2026-03-20 16:26:25.786479 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.786483 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.786487 | controller | - ' ' 2026-03-20 16:26:25.786491 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.786496 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.786500 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.786504 | controller | - ' exit 1' 2026-03-20 16:26:25.786508 | controller | - ' fi' 2026-03-20 16:26:25.786512 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.786517 | controller | - ' ' 2026-03-20 16:26:25.786521 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.786525 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.786529 | controller | | base64 -d)' 2026-03-20 16:26:25.786533 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.786538 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.786542 | controller | - ' exit 1' 2026-03-20 16:26:25.786546 | controller | - ' fi' 2026-03-20 16:26:25.786550 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.786554 | controller | - ' ' 2026-03-20 16:26:25.786558 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.786563 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.786567 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.786571 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.786577 | controller | --timeout=300s' 2026-03-20 16:26:25.786581 | controller | - ' ' 2026-03-20 16:26:25.786585 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.786589 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.786594 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.786598 | controller | - ' ' 2026-03-20 16:26:25.786602 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.786606 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.786610 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.786614 | controller | - ' ' 2026-03-20 16:26:25.786618 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.786623 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.786627 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.786631 | controller | available"' 2026-03-20 16:26:25.786643 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.786647 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.786651 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.786656 | controller | - ' exit 1' 2026-03-20 16:26:25.786660 | controller | - ' fi' 2026-03-20 16:26:25.786664 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.786668 | controller | - ' ' 2026-03-20 16:26:25.786673 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.786677 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.786681 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.786685 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.786689 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.786708 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.786713 | controller | -- \' 2026-03-20 16:26:25.786717 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.786723 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.786728 | controller | - ' logger.go:42: 16:14:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.786732 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.786740 | controller | - ' logger.go:42: 16:14:42 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.786744 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.786748 | controller | found' 2026-03-20 16:26:25.786755 | controller | - ' logger.go:42: 16:14:43 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.786759 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.786763 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.786768 | controller | - ' ' 2026-03-20 16:26:25.786772 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.786776 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.786782 | controller | - ' ' 2026-03-20 16:26:25.786786 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.786790 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.786795 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.786799 | controller | - ' exit 1' 2026-03-20 16:26:25.786803 | controller | - ' fi' 2026-03-20 16:26:25.786807 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.786811 | controller | - ' ' 2026-03-20 16:26:25.786817 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.786821 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.786826 | controller | | base64 -d)' 2026-03-20 16:26:25.786830 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.786834 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.786838 | controller | - ' exit 1' 2026-03-20 16:26:25.786842 | controller | - ' fi' 2026-03-20 16:26:25.786846 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.786851 | controller | - ' ' 2026-03-20 16:26:25.786855 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.786859 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.786865 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.786869 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.786873 | controller | --timeout=300s' 2026-03-20 16:26:25.786879 | controller | - ' ' 2026-03-20 16:26:25.786883 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.786887 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.786891 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.786896 | controller | - ' ' 2026-03-20 16:26:25.786900 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.786917 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.786922 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.786926 | controller | - ' ' 2026-03-20 16:26:25.786930 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.786934 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.786939 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.786943 | controller | available"' 2026-03-20 16:26:25.786947 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.786960 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.786965 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.786969 | controller | - ' exit 1' 2026-03-20 16:26:25.786973 | controller | - ' fi' 2026-03-20 16:26:25.786978 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.786982 | controller | - ' ' 2026-03-20 16:26:25.786986 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.786990 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.786994 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.787000 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.787005 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.787009 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.787013 | controller | -- \' 2026-03-20 16:26:25.787017 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.787021 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.787026 | controller | - ' logger.go:42: 16:14:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.787030 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.787034 | controller | - ' logger.go:42: 16:14:43 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.787038 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.787042 | controller | found' 2026-03-20 16:26:25.787047 | controller | - ' logger.go:42: 16:14:44 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.787051 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.787055 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.787059 | controller | - ' ' 2026-03-20 16:26:25.787064 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.787068 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.787072 | controller | - ' ' 2026-03-20 16:26:25.787076 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.787080 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.787084 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.787089 | controller | - ' exit 1' 2026-03-20 16:26:25.787093 | controller | - ' fi' 2026-03-20 16:26:25.787097 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.787101 | controller | - ' ' 2026-03-20 16:26:25.787105 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.787109 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.787114 | controller | | base64 -d)' 2026-03-20 16:26:25.787118 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.787122 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.787126 | controller | - ' exit 1' 2026-03-20 16:26:25.787130 | controller | - ' fi' 2026-03-20 16:26:25.787135 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.787139 | controller | - ' ' 2026-03-20 16:26:25.787143 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.787147 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.787151 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.787156 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.787160 | controller | --timeout=300s' 2026-03-20 16:26:25.787164 | controller | - ' ' 2026-03-20 16:26:25.787168 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.787172 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.787176 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.787181 | controller | - ' ' 2026-03-20 16:26:25.787185 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.787191 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.787195 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.787199 | controller | - ' ' 2026-03-20 16:26:25.787203 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.787207 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.787211 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.787216 | controller | available"' 2026-03-20 16:26:25.787220 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.787224 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.787236 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.787241 | controller | - ' exit 1' 2026-03-20 16:26:25.787245 | controller | - ' fi' 2026-03-20 16:26:25.787249 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.787253 | controller | - ' ' 2026-03-20 16:26:25.787257 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.787262 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.787266 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.787270 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.787274 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.787278 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.787283 | controller | -- \' 2026-03-20 16:26:25.787287 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.787291 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.787297 | controller | - ' logger.go:42: 16:14:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.787302 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.787308 | controller | - ' logger.go:42: 16:14:44 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.787314 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.787320 | controller | found' 2026-03-20 16:26:25.787326 | controller | - ' logger.go:42: 16:14:45 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.787332 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.787338 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.787343 | controller | - ' ' 2026-03-20 16:26:25.787349 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.787354 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.787360 | controller | - ' ' 2026-03-20 16:26:25.787366 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.787372 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.787377 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.787381 | controller | - ' exit 1' 2026-03-20 16:26:25.787385 | controller | - ' fi' 2026-03-20 16:26:25.787390 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.787394 | controller | - ' ' 2026-03-20 16:26:25.787398 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.787405 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.787409 | controller | | base64 -d)' 2026-03-20 16:26:25.787413 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.787418 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.787422 | controller | - ' exit 1' 2026-03-20 16:26:25.787426 | controller | - ' fi' 2026-03-20 16:26:25.787430 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.787434 | controller | - ' ' 2026-03-20 16:26:25.787439 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.787443 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.787447 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.787454 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.787458 | controller | --timeout=300s' 2026-03-20 16:26:25.787463 | controller | - ' ' 2026-03-20 16:26:25.787467 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.787471 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.787475 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.787479 | controller | - ' ' 2026-03-20 16:26:25.787483 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.787488 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.787492 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.787496 | controller | - ' ' 2026-03-20 16:26:25.787502 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.787519 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.787532 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.787537 | controller | available"' 2026-03-20 16:26:25.787541 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.787545 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.787549 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.787564 | controller | - ' exit 1' 2026-03-20 16:26:25.787569 | controller | - ' fi' 2026-03-20 16:26:25.787573 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.787577 | controller | - ' ' 2026-03-20 16:26:25.787582 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.787586 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.787590 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.787594 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.787598 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.787603 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.787607 | controller | -- \' 2026-03-20 16:26:25.787611 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.787620 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.787624 | controller | - ' logger.go:42: 16:14:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.787628 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.787632 | controller | - ' logger.go:42: 16:14:46 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.787637 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.787641 | controller | found' 2026-03-20 16:26:25.787645 | controller | - ' logger.go:42: 16:14:47 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.787649 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.787653 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.787658 | controller | - ' ' 2026-03-20 16:26:25.787662 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.787666 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.787670 | controller | - ' ' 2026-03-20 16:26:25.787674 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.787679 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.787683 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.787687 | controller | - ' exit 1' 2026-03-20 16:26:25.787726 | controller | - ' fi' 2026-03-20 16:26:25.787735 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.787740 | controller | - ' ' 2026-03-20 16:26:25.787744 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.787748 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.787752 | controller | | base64 -d)' 2026-03-20 16:26:25.787757 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.787761 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.787765 | controller | - ' exit 1' 2026-03-20 16:26:25.787769 | controller | - ' fi' 2026-03-20 16:26:25.787774 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.787778 | controller | - ' ' 2026-03-20 16:26:25.787782 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.787786 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.787790 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.787794 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.787799 | controller | --timeout=300s' 2026-03-20 16:26:25.787803 | controller | - ' ' 2026-03-20 16:26:25.787807 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.787811 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.787815 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.787819 | controller | - ' ' 2026-03-20 16:26:25.787824 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.787828 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.787832 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.787836 | controller | - ' ' 2026-03-20 16:26:25.787840 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.787845 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.787851 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.787855 | controller | available"' 2026-03-20 16:26:25.787860 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.787864 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.787868 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.787872 | controller | - ' exit 1' 2026-03-20 16:26:25.787876 | controller | - ' fi' 2026-03-20 16:26:25.787889 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.787894 | controller | - ' ' 2026-03-20 16:26:25.787898 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.787903 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.787907 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.787911 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.787915 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.787920 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.787926 | controller | -- \' 2026-03-20 16:26:25.787931 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.787935 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.787939 | controller | - ' logger.go:42: 16:14:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.787943 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.787947 | controller | - ' logger.go:42: 16:14:47 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.787952 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.787956 | controller | found' 2026-03-20 16:26:25.787960 | controller | - ' logger.go:42: 16:14:48 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.787964 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.787968 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.787973 | controller | - ' ' 2026-03-20 16:26:25.787977 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.787981 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.787985 | controller | - ' ' 2026-03-20 16:26:25.787989 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.787994 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.787998 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.788002 | controller | - ' exit 1' 2026-03-20 16:26:25.788006 | controller | - ' fi' 2026-03-20 16:26:25.788010 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.788015 | controller | - ' ' 2026-03-20 16:26:25.788021 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.788026 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.788030 | controller | | base64 -d)' 2026-03-20 16:26:25.788034 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.788039 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.788043 | controller | - ' exit 1' 2026-03-20 16:26:25.788047 | controller | - ' fi' 2026-03-20 16:26:25.788051 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.788059 | controller | - ' ' 2026-03-20 16:26:25.788063 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.788067 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.788072 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.788076 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.788080 | controller | --timeout=300s' 2026-03-20 16:26:25.788084 | controller | - ' ' 2026-03-20 16:26:25.788088 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.788093 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.788097 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.788101 | controller | - ' ' 2026-03-20 16:26:25.788105 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.788109 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.788114 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.788118 | controller | - ' ' 2026-03-20 16:26:25.788124 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.788128 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.788132 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.788136 | controller | available"' 2026-03-20 16:26:25.788141 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.788147 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.788151 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.788155 | controller | - ' exit 1' 2026-03-20 16:26:25.788159 | controller | - ' fi' 2026-03-20 16:26:25.788171 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.788176 | controller | - ' ' 2026-03-20 16:26:25.788180 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.788184 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.788188 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.788193 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.788197 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.788201 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.788205 | controller | -- \' 2026-03-20 16:26:25.788210 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.788214 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.788218 | controller | - ' logger.go:42: 16:14:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.788222 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.788226 | controller | - ' logger.go:42: 16:14:48 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.788231 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.788235 | controller | found' 2026-03-20 16:26:25.788242 | controller | - ' logger.go:42: 16:14:49 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.788247 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.788251 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.788255 | controller | - ' ' 2026-03-20 16:26:25.788259 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.788263 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.788267 | controller | - ' ' 2026-03-20 16:26:25.788272 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.788276 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.788280 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.788284 | controller | - ' exit 1' 2026-03-20 16:26:25.788288 | controller | - ' fi' 2026-03-20 16:26:25.788293 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.788297 | controller | - ' ' 2026-03-20 16:26:25.788301 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.788305 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.788309 | controller | | base64 -d)' 2026-03-20 16:26:25.788314 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.788318 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.788322 | controller | - ' exit 1' 2026-03-20 16:26:25.788326 | controller | - ' fi' 2026-03-20 16:26:25.788330 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.788335 | controller | - ' ' 2026-03-20 16:26:25.788339 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.788343 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.788347 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.788351 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.788355 | controller | --timeout=300s' 2026-03-20 16:26:25.788360 | controller | - ' ' 2026-03-20 16:26:25.788364 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.788368 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.788372 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.788376 | controller | - ' ' 2026-03-20 16:26:25.788381 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.788385 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.788389 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.788393 | controller | - ' ' 2026-03-20 16:26:25.788397 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.788402 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.788408 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.788419 | controller | available"' 2026-03-20 16:26:25.788432 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.788438 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.788444 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.788450 | controller | - ' exit 1' 2026-03-20 16:26:25.788455 | controller | - ' fi' 2026-03-20 16:26:25.788466 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.788471 | controller | - ' ' 2026-03-20 16:26:25.788490 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.788497 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.788503 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.788509 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.788515 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.788521 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.788526 | controller | -- \' 2026-03-20 16:26:25.788533 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.788537 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.788541 | controller | - ' logger.go:42: 16:14:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.788545 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.788550 | controller | - ' logger.go:42: 16:14:49 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.788554 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.788558 | controller | found' 2026-03-20 16:26:25.788562 | controller | - ' logger.go:42: 16:14:50 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.788566 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.788570 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.788574 | controller | - ' ' 2026-03-20 16:26:25.788579 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.788583 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.788587 | controller | - ' ' 2026-03-20 16:26:25.788591 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.788595 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.788599 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.788604 | controller | - ' exit 1' 2026-03-20 16:26:25.788608 | controller | - ' fi' 2026-03-20 16:26:25.788612 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.788616 | controller | - ' ' 2026-03-20 16:26:25.788620 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.788625 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.788629 | controller | | base64 -d)' 2026-03-20 16:26:25.788633 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.788637 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.788641 | controller | - ' exit 1' 2026-03-20 16:26:25.788645 | controller | - ' fi' 2026-03-20 16:26:25.788650 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.788654 | controller | - ' ' 2026-03-20 16:26:25.788658 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.788662 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.788666 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.788670 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.788674 | controller | --timeout=300s' 2026-03-20 16:26:25.788679 | controller | - ' ' 2026-03-20 16:26:25.788685 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.788690 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.788708 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.788712 | controller | - ' ' 2026-03-20 16:26:25.788716 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.788720 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.788725 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.788729 | controller | - ' ' 2026-03-20 16:26:25.788733 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.788737 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.788745 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.788749 | controller | available"' 2026-03-20 16:26:25.788754 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.788758 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.788762 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.788766 | controller | - ' exit 1' 2026-03-20 16:26:25.788770 | controller | - ' fi' 2026-03-20 16:26:25.788774 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.788779 | controller | - ' ' 2026-03-20 16:26:25.788791 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.788797 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.788801 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.788805 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.788809 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.788813 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.788818 | controller | -- \' 2026-03-20 16:26:25.788822 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.788826 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.788830 | controller | - ' logger.go:42: 16:14:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.788834 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.788838 | controller | - ' logger.go:42: 16:14:50 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.788842 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.788847 | controller | found' 2026-03-20 16:26:25.788851 | controller | - ' logger.go:42: 16:14:51 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.788855 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.788859 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.788863 | controller | - ' ' 2026-03-20 16:26:25.788867 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.788872 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.788876 | controller | - ' ' 2026-03-20 16:26:25.788880 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.788886 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.788891 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.788895 | controller | - ' exit 1' 2026-03-20 16:26:25.788899 | controller | - ' fi' 2026-03-20 16:26:25.788903 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.788908 | controller | - ' ' 2026-03-20 16:26:25.788912 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.788916 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.788920 | controller | | base64 -d)' 2026-03-20 16:26:25.788924 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.788928 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.788932 | controller | - ' exit 1' 2026-03-20 16:26:25.788937 | controller | - ' fi' 2026-03-20 16:26:25.788941 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.788948 | controller | - ' ' 2026-03-20 16:26:25.788952 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.788956 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.788960 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.788965 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.788969 | controller | --timeout=300s' 2026-03-20 16:26:25.788973 | controller | - ' ' 2026-03-20 16:26:25.788977 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.788981 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.788985 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.788990 | controller | - ' ' 2026-03-20 16:26:25.788994 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.788998 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.789004 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.789008 | controller | - ' ' 2026-03-20 16:26:25.789013 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.789017 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.789021 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.789025 | controller | available"' 2026-03-20 16:26:25.789029 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.789033 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.789037 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.789042 | controller | - ' exit 1' 2026-03-20 16:26:25.789046 | controller | - ' fi' 2026-03-20 16:26:25.789050 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.789054 | controller | - ' ' 2026-03-20 16:26:25.789060 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.789073 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.789078 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.789082 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.789088 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.789093 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.789097 | controller | -- \' 2026-03-20 16:26:25.789101 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.789105 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.789109 | controller | - ' logger.go:42: 16:14:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.789114 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.789118 | controller | - ' logger.go:42: 16:14:51 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.789122 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.789126 | controller | found' 2026-03-20 16:26:25.789130 | controller | - ' logger.go:42: 16:14:52 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.789135 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.789139 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.789143 | controller | - ' ' 2026-03-20 16:26:25.789147 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.789152 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.789156 | controller | - ' ' 2026-03-20 16:26:25.789160 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.789164 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.789168 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.789175 | controller | - ' exit 1' 2026-03-20 16:26:25.789179 | controller | - ' fi' 2026-03-20 16:26:25.789184 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.789188 | controller | - ' ' 2026-03-20 16:26:25.789192 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.789196 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.789200 | controller | | base64 -d)' 2026-03-20 16:26:25.789205 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.789209 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.789213 | controller | - ' exit 1' 2026-03-20 16:26:25.789217 | controller | - ' fi' 2026-03-20 16:26:25.789221 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.789226 | controller | - ' ' 2026-03-20 16:26:25.789230 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.789234 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.789238 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.789242 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.789246 | controller | --timeout=300s' 2026-03-20 16:26:25.789252 | controller | - ' ' 2026-03-20 16:26:25.789257 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.789261 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.789265 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.789269 | controller | - ' ' 2026-03-20 16:26:25.789273 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.789279 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.789284 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.789288 | controller | - ' ' 2026-03-20 16:26:25.789292 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.789296 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.789300 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.789305 | controller | available"' 2026-03-20 16:26:25.789309 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.789313 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.789317 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.789321 | controller | - ' exit 1' 2026-03-20 16:26:25.789326 | controller | - ' fi' 2026-03-20 16:26:25.789330 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.789334 | controller | - ' ' 2026-03-20 16:26:25.789338 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.789342 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.789355 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.789359 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.789363 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.789368 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.789373 | controller | -- \' 2026-03-20 16:26:25.789379 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.789394 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.789403 | controller | - ' logger.go:42: 16:14:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.789409 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.789422 | controller | - ' logger.go:42: 16:14:52 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.789428 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.789434 | controller | found' 2026-03-20 16:26:25.789440 | controller | - ' logger.go:42: 16:14:53 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.789444 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.789448 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.789452 | controller | - ' ' 2026-03-20 16:26:25.789457 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.789461 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.789465 | controller | - ' ' 2026-03-20 16:26:25.789469 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.789473 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.789478 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.789482 | controller | - ' exit 1' 2026-03-20 16:26:25.789486 | controller | - ' fi' 2026-03-20 16:26:25.789490 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.789494 | controller | - ' ' 2026-03-20 16:26:25.789501 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.789509 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.789513 | controller | | base64 -d)' 2026-03-20 16:26:25.789517 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.789522 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.789526 | controller | - ' exit 1' 2026-03-20 16:26:25.789530 | controller | - ' fi' 2026-03-20 16:26:25.789534 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.789539 | controller | - ' ' 2026-03-20 16:26:25.789543 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.789547 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.789551 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.789555 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.789560 | controller | --timeout=300s' 2026-03-20 16:26:25.789564 | controller | - ' ' 2026-03-20 16:26:25.789568 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.789572 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.789576 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.789581 | controller | - ' ' 2026-03-20 16:26:25.789585 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.789589 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.789593 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.789597 | controller | - ' ' 2026-03-20 16:26:25.789602 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.789606 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.789610 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.789614 | controller | available"' 2026-03-20 16:26:25.789619 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.789623 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.789627 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.789631 | controller | - ' exit 1' 2026-03-20 16:26:25.789635 | controller | - ' fi' 2026-03-20 16:26:25.789640 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.789644 | controller | - ' ' 2026-03-20 16:26:25.789648 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.789652 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.789668 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.789675 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.789680 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.789685 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.789690 | controller | -- \' 2026-03-20 16:26:25.789727 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.789732 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.789738 | controller | - ' logger.go:42: 16:14:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.789743 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.789747 | controller | - ' logger.go:42: 16:14:54 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.789751 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.789755 | controller | found' 2026-03-20 16:26:25.789759 | controller | - ' logger.go:42: 16:14:55 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.789764 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.789768 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.789772 | controller | - ' ' 2026-03-20 16:26:25.789776 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.789781 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.789785 | controller | - ' ' 2026-03-20 16:26:25.789789 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.789793 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.789797 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.789802 | controller | - ' exit 1' 2026-03-20 16:26:25.789806 | controller | - ' fi' 2026-03-20 16:26:25.789810 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.789814 | controller | - ' ' 2026-03-20 16:26:25.789819 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.789823 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.789827 | controller | | base64 -d)' 2026-03-20 16:26:25.789831 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.789836 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.789840 | controller | - ' exit 1' 2026-03-20 16:26:25.789844 | controller | - ' fi' 2026-03-20 16:26:25.789848 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.789852 | controller | - ' ' 2026-03-20 16:26:25.789857 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.789861 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.789865 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.789869 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.789873 | controller | --timeout=300s' 2026-03-20 16:26:25.789877 | controller | - ' ' 2026-03-20 16:26:25.789882 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.789887 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.789893 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.789897 | controller | - ' ' 2026-03-20 16:26:25.789901 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.789905 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.789910 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.789914 | controller | - ' ' 2026-03-20 16:26:25.789918 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.789922 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.789928 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.789933 | controller | available"' 2026-03-20 16:26:25.789937 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.789941 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.789945 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.789950 | controller | - ' exit 1' 2026-03-20 16:26:25.789954 | controller | - ' fi' 2026-03-20 16:26:25.789958 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.789962 | controller | - ' ' 2026-03-20 16:26:25.789969 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.789973 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.789986 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.789991 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.789995 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.789999 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.790004 | controller | -- \' 2026-03-20 16:26:25.790008 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.790012 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.790018 | controller | - ' logger.go:42: 16:14:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.790023 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.790027 | controller | - ' logger.go:42: 16:14:55 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.790031 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.790035 | controller | found' 2026-03-20 16:26:25.790040 | controller | - ' logger.go:42: 16:14:56 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.790044 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.790048 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.790053 | controller | - ' ' 2026-03-20 16:26:25.790057 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.790061 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.790065 | controller | - ' ' 2026-03-20 16:26:25.790069 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.790073 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.790078 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.790082 | controller | - ' exit 1' 2026-03-20 16:26:25.790086 | controller | - ' fi' 2026-03-20 16:26:25.790090 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.790094 | controller | - ' ' 2026-03-20 16:26:25.790099 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.790103 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.790107 | controller | | base64 -d)' 2026-03-20 16:26:25.790111 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.790115 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.790120 | controller | - ' exit 1' 2026-03-20 16:26:25.790124 | controller | - ' fi' 2026-03-20 16:26:25.790128 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.790132 | controller | - ' ' 2026-03-20 16:26:25.790152 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.790157 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.790161 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.790165 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.790169 | controller | --timeout=300s' 2026-03-20 16:26:25.790173 | controller | - ' ' 2026-03-20 16:26:25.790178 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.790182 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.790186 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.790190 | controller | - ' ' 2026-03-20 16:26:25.790197 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.790201 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.790205 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.790210 | controller | - ' ' 2026-03-20 16:26:25.790214 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.790218 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.790222 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.790226 | controller | available"' 2026-03-20 16:26:25.790231 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.790235 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.790239 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.790243 | controller | - ' exit 1' 2026-03-20 16:26:25.790247 | controller | - ' fi' 2026-03-20 16:26:25.790252 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.790256 | controller | - ' ' 2026-03-20 16:26:25.790260 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.790264 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.790268 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.790281 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.790286 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.790291 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.790295 | controller | -- \' 2026-03-20 16:26:25.790299 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.790304 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.790308 | controller | - ' logger.go:42: 16:14:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.790312 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.790317 | controller | - ' logger.go:42: 16:14:56 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.790321 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.790325 | controller | found' 2026-03-20 16:26:25.790330 | controller | - ' logger.go:42: 16:14:57 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.790336 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.790340 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.790347 | controller | - ' ' 2026-03-20 16:26:25.790351 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.790355 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.790360 | controller | - ' ' 2026-03-20 16:26:25.790364 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.790368 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.790373 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.790377 | controller | - ' exit 1' 2026-03-20 16:26:25.790382 | controller | - ' fi' 2026-03-20 16:26:25.790386 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.790390 | controller | - ' ' 2026-03-20 16:26:25.790395 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.790400 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.790405 | controller | | base64 -d)' 2026-03-20 16:26:25.790409 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.790414 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.790418 | controller | - ' exit 1' 2026-03-20 16:26:25.790422 | controller | - ' fi' 2026-03-20 16:26:25.790427 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.790431 | controller | - ' ' 2026-03-20 16:26:25.790435 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.790439 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.790444 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.790448 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.790452 | controller | --timeout=300s' 2026-03-20 16:26:25.790456 | controller | - ' ' 2026-03-20 16:26:25.790461 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.790465 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.790469 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.790474 | controller | - ' ' 2026-03-20 16:26:25.790478 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.790483 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.790488 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.790492 | controller | - ' ' 2026-03-20 16:26:25.790496 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.790501 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.790505 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.790509 | controller | available"' 2026-03-20 16:26:25.790514 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.790518 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.790522 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.790526 | controller | - ' exit 1' 2026-03-20 16:26:25.790531 | controller | - ' fi' 2026-03-20 16:26:25.790537 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.790551 | controller | - ' ' 2026-03-20 16:26:25.790564 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.790571 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.790577 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.790588 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.790604 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.790610 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.790614 | controller | -- \' 2026-03-20 16:26:25.790619 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.790624 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.790628 | controller | - ' logger.go:42: 16:14:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.790632 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.790637 | controller | - ' logger.go:42: 16:14:57 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.790641 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.790645 | controller | found' 2026-03-20 16:26:25.790650 | controller | - ' logger.go:42: 16:14:58 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.790654 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.790658 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.790663 | controller | - ' ' 2026-03-20 16:26:25.790667 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.790671 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.790676 | controller | - ' ' 2026-03-20 16:26:25.790680 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.790684 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.790688 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.790706 | controller | - ' exit 1' 2026-03-20 16:26:25.790716 | controller | - ' fi' 2026-03-20 16:26:25.790721 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.790725 | controller | - ' ' 2026-03-20 16:26:25.790729 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.790734 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.790738 | controller | | base64 -d)' 2026-03-20 16:26:25.790742 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.790748 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.790755 | controller | - ' exit 1' 2026-03-20 16:26:25.790761 | controller | - ' fi' 2026-03-20 16:26:25.790767 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.790773 | controller | - ' ' 2026-03-20 16:26:25.790779 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.790785 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.790791 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.790797 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.790803 | controller | --timeout=300s' 2026-03-20 16:26:25.790809 | controller | - ' ' 2026-03-20 16:26:25.790813 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.790821 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.790825 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.790829 | controller | - ' ' 2026-03-20 16:26:25.790834 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.790838 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.790842 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.790846 | controller | - ' ' 2026-03-20 16:26:25.790851 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.790855 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.790859 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.790863 | controller | available"' 2026-03-20 16:26:25.790868 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.790876 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.790881 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.790885 | controller | - ' exit 1' 2026-03-20 16:26:25.790889 | controller | - ' fi' 2026-03-20 16:26:25.790894 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.790898 | controller | - ' ' 2026-03-20 16:26:25.790902 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.790907 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.790911 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.790915 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.790929 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.790934 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.790939 | controller | -- \' 2026-03-20 16:26:25.790943 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.790947 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.790952 | controller | - ' logger.go:42: 16:14:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.790957 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.790961 | controller | - ' logger.go:42: 16:14:58 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.790966 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.790970 | controller | found' 2026-03-20 16:26:25.790974 | controller | - ' logger.go:42: 16:14:59 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.790978 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.790983 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.790987 | controller | - ' ' 2026-03-20 16:26:25.790991 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.790995 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.791000 | controller | - ' ' 2026-03-20 16:26:25.791004 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.791010 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.791015 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.791019 | controller | - ' exit 1' 2026-03-20 16:26:25.791023 | controller | - ' fi' 2026-03-20 16:26:25.791028 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.791032 | controller | - ' ' 2026-03-20 16:26:25.791036 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.791040 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.791045 | controller | | base64 -d)' 2026-03-20 16:26:25.791049 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.791053 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.791058 | controller | - ' exit 1' 2026-03-20 16:26:25.791062 | controller | - ' fi' 2026-03-20 16:26:25.791066 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.791070 | controller | - ' ' 2026-03-20 16:26:25.791075 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.791079 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.791083 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.791087 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.791092 | controller | --timeout=300s' 2026-03-20 16:26:25.791096 | controller | - ' ' 2026-03-20 16:26:25.791100 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.791105 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.791109 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.791113 | controller | - ' ' 2026-03-20 16:26:25.791117 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.791122 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.791126 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.791130 | controller | - ' ' 2026-03-20 16:26:25.791135 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.791141 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.791145 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.791149 | controller | available"' 2026-03-20 16:26:25.791154 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.791158 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.791162 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.791166 | controller | - ' exit 1' 2026-03-20 16:26:25.791170 | controller | - ' fi' 2026-03-20 16:26:25.791175 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.791179 | controller | - ' ' 2026-03-20 16:26:25.791183 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.791187 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.791191 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.791196 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.791211 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.791217 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.791221 | controller | -- \' 2026-03-20 16:26:25.791225 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.791230 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.791234 | controller | - ' logger.go:42: 16:14:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.791238 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.791242 | controller | - ' logger.go:42: 16:14:59 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.791247 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.791251 | controller | found' 2026-03-20 16:26:25.791255 | controller | - ' logger.go:42: 16:15:00 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.791259 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.791263 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.791268 | controller | - ' ' 2026-03-20 16:26:25.791272 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.791276 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.791280 | controller | - ' ' 2026-03-20 16:26:25.791284 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.791289 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.791293 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.791297 | controller | - ' exit 1' 2026-03-20 16:26:25.791301 | controller | - ' fi' 2026-03-20 16:26:25.791305 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.791309 | controller | - ' ' 2026-03-20 16:26:25.791314 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.791318 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.791322 | controller | | base64 -d)' 2026-03-20 16:26:25.791326 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.791330 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.791335 | controller | - ' exit 1' 2026-03-20 16:26:25.791341 | controller | - ' fi' 2026-03-20 16:26:25.791345 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.791350 | controller | - ' ' 2026-03-20 16:26:25.791354 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.791358 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.791362 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.791366 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.791371 | controller | --timeout=300s' 2026-03-20 16:26:25.791375 | controller | - ' ' 2026-03-20 16:26:25.791379 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.791383 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.791388 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.791392 | controller | - ' ' 2026-03-20 16:26:25.791396 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.791400 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.791406 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.791410 | controller | - ' ' 2026-03-20 16:26:25.791414 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.791419 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.791424 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.791428 | controller | available"' 2026-03-20 16:26:25.791432 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.791436 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.791441 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.791446 | controller | - ' exit 1' 2026-03-20 16:26:25.791451 | controller | - ' fi' 2026-03-20 16:26:25.791456 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.791461 | controller | - ' ' 2026-03-20 16:26:25.791465 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.791469 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.791473 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.791477 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.791482 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.791494 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.791499 | controller | -- \' 2026-03-20 16:26:25.791503 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.791507 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.791512 | controller | - ' logger.go:42: 16:15:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.791516 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.791520 | controller | - ' logger.go:42: 16:15:00 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.791524 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.791529 | controller | found' 2026-03-20 16:26:25.791533 | controller | - ' logger.go:42: 16:15:01 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.791537 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.791541 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.791545 | controller | - ' ' 2026-03-20 16:26:25.791550 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.791554 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.791558 | controller | - ' ' 2026-03-20 16:26:25.791562 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.791566 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.791571 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.791575 | controller | - ' exit 1' 2026-03-20 16:26:25.791579 | controller | - ' fi' 2026-03-20 16:26:25.791583 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.791587 | controller | - ' ' 2026-03-20 16:26:25.791592 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.791597 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.791605 | controller | | base64 -d)' 2026-03-20 16:26:25.791609 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.791613 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.791617 | controller | - ' exit 1' 2026-03-20 16:26:25.791623 | controller | - ' fi' 2026-03-20 16:26:25.791629 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.791635 | controller | - ' ' 2026-03-20 16:26:25.791641 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.791647 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.791653 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.791662 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.791668 | controller | --timeout=300s' 2026-03-20 16:26:25.791674 | controller | - ' ' 2026-03-20 16:26:25.791679 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.791685 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.791691 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.791819 | controller | - ' ' 2026-03-20 16:26:25.791824 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.791829 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.791833 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.791837 | controller | - ' ' 2026-03-20 16:26:25.791841 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.791845 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.791854 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.791859 | controller | available"' 2026-03-20 16:26:25.791863 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.791867 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.791871 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.791875 | controller | - ' exit 1' 2026-03-20 16:26:25.791880 | controller | - ' fi' 2026-03-20 16:26:25.791884 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.791888 | controller | - ' ' 2026-03-20 16:26:25.791892 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.791896 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.791901 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.791905 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.791909 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.791923 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.791929 | controller | -- \' 2026-03-20 16:26:25.791934 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.791938 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.791942 | controller | - ' logger.go:42: 16:15:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.791949 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.791956 | controller | - ' logger.go:42: 16:15:02 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.791960 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.791964 | controller | found' 2026-03-20 16:26:25.791969 | controller | - ' logger.go:42: 16:15:03 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.791973 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.791977 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.791981 | controller | - ' ' 2026-03-20 16:26:25.791985 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.791990 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.791994 | controller | - ' ' 2026-03-20 16:26:25.791998 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.792002 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.792006 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.792011 | controller | - ' exit 1' 2026-03-20 16:26:25.792015 | controller | - ' fi' 2026-03-20 16:26:25.792019 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.792023 | controller | - ' ' 2026-03-20 16:26:25.792028 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.792032 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.792036 | controller | | base64 -d)' 2026-03-20 16:26:25.792040 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.792045 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.792049 | controller | - ' exit 1' 2026-03-20 16:26:25.792053 | controller | - ' fi' 2026-03-20 16:26:25.792057 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.792061 | controller | - ' ' 2026-03-20 16:26:25.792066 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.792070 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.792074 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.792078 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.792083 | controller | --timeout=300s' 2026-03-20 16:26:25.792087 | controller | - ' ' 2026-03-20 16:26:25.792091 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.792095 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.792099 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.792104 | controller | - ' ' 2026-03-20 16:26:25.792108 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.792112 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.792116 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.792120 | controller | - ' ' 2026-03-20 16:26:25.792125 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.792129 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.792133 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.792139 | controller | available"' 2026-03-20 16:26:25.792143 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.792147 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.792152 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.792156 | controller | - ' exit 1' 2026-03-20 16:26:25.792160 | controller | - ' fi' 2026-03-20 16:26:25.792164 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.792168 | controller | - ' ' 2026-03-20 16:26:25.792173 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.792177 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.792181 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.792185 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.792189 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.792194 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.792198 | controller | -- \' 2026-03-20 16:26:25.792211 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.792216 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.792220 | controller | - ' logger.go:42: 16:15:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.792224 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.792228 | controller | - ' logger.go:42: 16:15:03 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.792233 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.792237 | controller | found' 2026-03-20 16:26:25.792241 | controller | - ' logger.go:42: 16:15:04 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.792245 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.792249 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.792254 | controller | - ' ' 2026-03-20 16:26:25.792258 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.792262 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.792266 | controller | - ' ' 2026-03-20 16:26:25.792270 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.792275 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.792279 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.792283 | controller | - ' exit 1' 2026-03-20 16:26:25.792287 | controller | - ' fi' 2026-03-20 16:26:25.792292 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.792296 | controller | - ' ' 2026-03-20 16:26:25.792302 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.792308 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.792313 | controller | | base64 -d)' 2026-03-20 16:26:25.792317 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.792322 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.792327 | controller | - ' exit 1' 2026-03-20 16:26:25.792332 | controller | - ' fi' 2026-03-20 16:26:25.792336 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.792340 | controller | - ' ' 2026-03-20 16:26:25.792345 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.792352 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.792356 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.792360 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.792364 | controller | --timeout=300s' 2026-03-20 16:26:25.792368 | controller | - ' ' 2026-03-20 16:26:25.792373 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.792377 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.792385 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.792389 | controller | - ' ' 2026-03-20 16:26:25.792393 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.792397 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.792402 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.792406 | controller | - ' ' 2026-03-20 16:26:25.792410 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.792414 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.792418 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.792423 | controller | available"' 2026-03-20 16:26:25.792427 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.792431 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.792435 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.792439 | controller | - ' exit 1' 2026-03-20 16:26:25.792443 | controller | - ' fi' 2026-03-20 16:26:25.792448 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.792452 | controller | - ' ' 2026-03-20 16:26:25.792456 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.792460 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.792464 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.792468 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.792473 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.792477 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.792481 | controller | -- \' 2026-03-20 16:26:25.792493 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.792497 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.792502 | controller | - ' logger.go:42: 16:15:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.792506 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.792510 | controller | - ' logger.go:42: 16:15:04 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.792516 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.792521 | controller | found' 2026-03-20 16:26:25.792525 | controller | - ' logger.go:42: 16:15:05 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.792529 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.792536 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.792540 | controller | - ' ' 2026-03-20 16:26:25.792544 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.792548 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.792553 | controller | - ' ' 2026-03-20 16:26:25.792559 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.792563 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.792567 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.792571 | controller | - ' exit 1' 2026-03-20 16:26:25.792576 | controller | - ' fi' 2026-03-20 16:26:25.792580 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.792584 | controller | - ' ' 2026-03-20 16:26:25.792588 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.792592 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.792597 | controller | | base64 -d)' 2026-03-20 16:26:25.792601 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.792605 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.792609 | controller | - ' exit 1' 2026-03-20 16:26:25.792613 | controller | - ' fi' 2026-03-20 16:26:25.792618 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.792622 | controller | - ' ' 2026-03-20 16:26:25.792626 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.792630 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.792634 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.792638 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.792643 | controller | --timeout=300s' 2026-03-20 16:26:25.792647 | controller | - ' ' 2026-03-20 16:26:25.792651 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.792657 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.792668 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.792681 | controller | - ' ' 2026-03-20 16:26:25.792687 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.792710 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.792719 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.792725 | controller | - ' ' 2026-03-20 16:26:25.792730 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.792736 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.792741 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.792747 | controller | available"' 2026-03-20 16:26:25.792753 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.792759 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.792765 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.792770 | controller | - ' exit 1' 2026-03-20 16:26:25.792776 | controller | - ' fi' 2026-03-20 16:26:25.792781 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.792790 | controller | - ' ' 2026-03-20 16:26:25.792796 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.792801 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.792807 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.792812 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.792816 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.792820 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.792824 | controller | -- \' 2026-03-20 16:26:25.792828 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.792843 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.792849 | controller | - ' logger.go:42: 16:15:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.792864 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.792874 | controller | - ' logger.go:42: 16:15:05 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.792880 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.792886 | controller | found' 2026-03-20 16:26:25.792892 | controller | - ' logger.go:42: 16:15:06 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.792898 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.792903 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.792909 | controller | - ' ' 2026-03-20 16:26:25.792915 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.792920 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.792926 | controller | - ' ' 2026-03-20 16:26:25.792931 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.792935 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.792939 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.792943 | controller | - ' exit 1' 2026-03-20 16:26:25.792947 | controller | - ' fi' 2026-03-20 16:26:25.792951 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.792956 | controller | - ' ' 2026-03-20 16:26:25.792960 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.792964 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.792968 | controller | | base64 -d)' 2026-03-20 16:26:25.792972 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.792976 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.792980 | controller | - ' exit 1' 2026-03-20 16:26:25.792985 | controller | - ' fi' 2026-03-20 16:26:25.792989 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.792993 | controller | - ' ' 2026-03-20 16:26:25.792997 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.793001 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.793005 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.793010 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.793014 | controller | --timeout=300s' 2026-03-20 16:26:25.793018 | controller | - ' ' 2026-03-20 16:26:25.793022 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.793029 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.793034 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.793038 | controller | - ' ' 2026-03-20 16:26:25.793042 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.793046 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.793050 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.793054 | controller | - ' ' 2026-03-20 16:26:25.793058 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.793063 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.793067 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.793071 | controller | available"' 2026-03-20 16:26:25.793075 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.793079 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.793083 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.793088 | controller | - ' exit 1' 2026-03-20 16:26:25.793092 | controller | - ' fi' 2026-03-20 16:26:25.793096 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.793100 | controller | - ' ' 2026-03-20 16:26:25.793104 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.793108 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.793112 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.793117 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.793121 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.793125 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.793129 | controller | -- \' 2026-03-20 16:26:25.793133 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.793149 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.793154 | controller | - ' logger.go:42: 16:15:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.793158 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.793163 | controller | - ' logger.go:42: 16:15:06 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.793167 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.793171 | controller | found' 2026-03-20 16:26:25.793178 | controller | - ' logger.go:42: 16:15:07 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.793183 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.793187 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.793191 | controller | - ' ' 2026-03-20 16:26:25.793195 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.793199 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.793204 | controller | - ' ' 2026-03-20 16:26:25.793208 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.793212 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.793219 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.793223 | controller | - ' exit 1' 2026-03-20 16:26:25.793227 | controller | - ' fi' 2026-03-20 16:26:25.793232 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.793236 | controller | - ' ' 2026-03-20 16:26:25.793240 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.793244 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.793248 | controller | | base64 -d)' 2026-03-20 16:26:25.793252 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.793257 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.793261 | controller | - ' exit 1' 2026-03-20 16:26:25.793265 | controller | - ' fi' 2026-03-20 16:26:25.793269 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.793284 | controller | - ' ' 2026-03-20 16:26:25.793288 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.793292 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.793296 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.793301 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.793305 | controller | --timeout=300s' 2026-03-20 16:26:25.793309 | controller | - ' ' 2026-03-20 16:26:25.793313 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.793317 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.793321 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.793326 | controller | - ' ' 2026-03-20 16:26:25.793330 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.793334 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.793338 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.793342 | controller | - ' ' 2026-03-20 16:26:25.793346 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.793350 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.793355 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.793359 | controller | available"' 2026-03-20 16:26:25.793363 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.793367 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.793371 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.793375 | controller | - ' exit 1' 2026-03-20 16:26:25.793380 | controller | - ' fi' 2026-03-20 16:26:25.793384 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.793388 | controller | - ' ' 2026-03-20 16:26:25.793392 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.793396 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.793400 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.793405 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.793411 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.793417 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.793421 | controller | -- \' 2026-03-20 16:26:25.793425 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.793429 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.793442 | controller | - ' logger.go:42: 16:15:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.793447 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.793451 | controller | - ' logger.go:42: 16:15:07 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.793455 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.793459 | controller | found' 2026-03-20 16:26:25.793464 | controller | - ' logger.go:42: 16:15:08 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.793468 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.793472 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.793476 | controller | - ' ' 2026-03-20 16:26:25.793480 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.793484 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.793488 | controller | - ' ' 2026-03-20 16:26:25.793493 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.793497 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.793501 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.793505 | controller | - ' exit 1' 2026-03-20 16:26:25.793509 | controller | - ' fi' 2026-03-20 16:26:25.793513 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.793518 | controller | - ' ' 2026-03-20 16:26:25.793522 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.793526 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.793530 | controller | | base64 -d)' 2026-03-20 16:26:25.793534 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.793538 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.793542 | controller | - ' exit 1' 2026-03-20 16:26:25.793547 | controller | - ' fi' 2026-03-20 16:26:25.793551 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.793555 | controller | - ' ' 2026-03-20 16:26:25.793559 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.793563 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.793568 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.793572 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.793576 | controller | --timeout=300s' 2026-03-20 16:26:25.793582 | controller | - ' ' 2026-03-20 16:26:25.793586 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.793590 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.793594 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.793599 | controller | - ' ' 2026-03-20 16:26:25.793603 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.793607 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.793613 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.793617 | controller | - ' ' 2026-03-20 16:26:25.793621 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.793625 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.793630 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.793634 | controller | available"' 2026-03-20 16:26:25.793638 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.793642 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.793648 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.793652 | controller | - ' exit 1' 2026-03-20 16:26:25.793658 | controller | - ' fi' 2026-03-20 16:26:25.793662 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.793667 | controller | - ' ' 2026-03-20 16:26:25.793671 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.793675 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.793679 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.793683 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.793687 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.793706 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.793714 | controller | -- \' 2026-03-20 16:26:25.793718 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.793722 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.793734 | controller | - ' logger.go:42: 16:15:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.793739 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.793743 | controller | - ' logger.go:42: 16:15:09 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.793747 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.793752 | controller | found' 2026-03-20 16:26:25.793756 | controller | - ' logger.go:42: 16:15:10 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.793761 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.793767 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.793778 | controller | - ' ' 2026-03-20 16:26:25.793791 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.793797 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.793803 | controller | - ' ' 2026-03-20 16:26:25.793809 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.793814 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.793820 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.793826 | controller | - ' exit 1' 2026-03-20 16:26:25.793838 | controller | - ' fi' 2026-03-20 16:26:25.793844 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.793850 | controller | - ' ' 2026-03-20 16:26:25.793856 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.793861 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.793867 | controller | | base64 -d)' 2026-03-20 16:26:25.793877 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.793883 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.793889 | controller | - ' exit 1' 2026-03-20 16:26:25.793894 | controller | - ' fi' 2026-03-20 16:26:25.793898 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.793903 | controller | - ' ' 2026-03-20 16:26:25.793907 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.793911 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.793915 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.793919 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.793924 | controller | --timeout=300s' 2026-03-20 16:26:25.793928 | controller | - ' ' 2026-03-20 16:26:25.793932 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.793936 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.793940 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.793945 | controller | - ' ' 2026-03-20 16:26:25.793949 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.793953 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.793957 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.793961 | controller | - ' ' 2026-03-20 16:26:25.793966 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.793970 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.793974 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.793978 | controller | available"' 2026-03-20 16:26:25.793982 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.793986 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.793991 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.793995 | controller | - ' exit 1' 2026-03-20 16:26:25.793999 | controller | - ' fi' 2026-03-20 16:26:25.794003 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.794007 | controller | - ' ' 2026-03-20 16:26:25.794011 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.794016 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.794020 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.794024 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.794028 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.794032 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.794037 | controller | -- \' 2026-03-20 16:26:25.794041 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.794045 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.794049 | controller | - ' logger.go:42: 16:15:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.794056 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.794072 | controller | - ' logger.go:42: 16:15:10 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.794077 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.794082 | controller | found' 2026-03-20 16:26:25.794086 | controller | - ' logger.go:42: 16:15:11 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.794090 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.794094 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.794099 | controller | - ' ' 2026-03-20 16:26:25.794103 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.794107 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.794111 | controller | - ' ' 2026-03-20 16:26:25.794115 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.794119 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.794124 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.794128 | controller | - ' exit 1' 2026-03-20 16:26:25.794132 | controller | - ' fi' 2026-03-20 16:26:25.794136 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.794140 | controller | - ' ' 2026-03-20 16:26:25.794145 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.794149 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.794153 | controller | | base64 -d)' 2026-03-20 16:26:25.794157 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.794161 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.794166 | controller | - ' exit 1' 2026-03-20 16:26:25.794170 | controller | - ' fi' 2026-03-20 16:26:25.794174 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.794178 | controller | - ' ' 2026-03-20 16:26:25.794183 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.794187 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.794191 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.794195 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.794199 | controller | --timeout=300s' 2026-03-20 16:26:25.794204 | controller | - ' ' 2026-03-20 16:26:25.794208 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.794212 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.794216 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.794220 | controller | - ' ' 2026-03-20 16:26:25.794225 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.794229 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.794233 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.794237 | controller | - ' ' 2026-03-20 16:26:25.794241 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.794245 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.794250 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.794254 | controller | available"' 2026-03-20 16:26:25.794260 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.794268 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.794272 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.794276 | controller | - ' exit 1' 2026-03-20 16:26:25.794280 | controller | - ' fi' 2026-03-20 16:26:25.794285 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.794289 | controller | - ' ' 2026-03-20 16:26:25.794293 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.794297 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.794302 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.794306 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.794310 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.794314 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.794318 | controller | -- \' 2026-03-20 16:26:25.794323 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.794327 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.794331 | controller | - ' logger.go:42: 16:15:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.794335 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.794349 | controller | - ' logger.go:42: 16:15:11 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.794354 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.794358 | controller | found' 2026-03-20 16:26:25.794363 | controller | - ' logger.go:42: 16:15:12 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.794367 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.794372 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.794376 | controller | - ' ' 2026-03-20 16:26:25.794380 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.794394 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.794399 | controller | - ' ' 2026-03-20 16:26:25.794404 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.794408 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.794412 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.794417 | controller | - ' exit 1' 2026-03-20 16:26:25.794421 | controller | - ' fi' 2026-03-20 16:26:25.794425 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.794429 | controller | - ' ' 2026-03-20 16:26:25.794433 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.794438 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.794442 | controller | | base64 -d)' 2026-03-20 16:26:25.794446 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.794450 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.794454 | controller | - ' exit 1' 2026-03-20 16:26:25.794458 | controller | - ' fi' 2026-03-20 16:26:25.794463 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.794467 | controller | - ' ' 2026-03-20 16:26:25.794471 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.794475 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.794482 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.794486 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.794490 | controller | --timeout=300s' 2026-03-20 16:26:25.794494 | controller | - ' ' 2026-03-20 16:26:25.794498 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.794503 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.794507 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.794511 | controller | - ' ' 2026-03-20 16:26:25.794515 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.794521 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.794525 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.794529 | controller | - ' ' 2026-03-20 16:26:25.794533 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.794538 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.794542 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.794546 | controller | available"' 2026-03-20 16:26:25.794550 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.794554 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.794559 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.794563 | controller | - ' exit 1' 2026-03-20 16:26:25.794567 | controller | - ' fi' 2026-03-20 16:26:25.794571 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.794575 | controller | - ' ' 2026-03-20 16:26:25.794580 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.794584 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.794588 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.794592 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.794596 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.794600 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.794605 | controller | -- \' 2026-03-20 16:26:25.794609 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.794613 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.794617 | controller | - ' logger.go:42: 16:15:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.794621 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.794625 | controller | - ' logger.go:42: 16:15:12 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.794637 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.794642 | controller | found' 2026-03-20 16:26:25.794646 | controller | - ' logger.go:42: 16:15:13 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.794650 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.794655 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.794660 | controller | - ' ' 2026-03-20 16:26:25.794665 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.794669 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.794673 | controller | - ' ' 2026-03-20 16:26:25.794677 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.794681 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.794685 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.794705 | controller | - ' exit 1' 2026-03-20 16:26:25.794713 | controller | - ' fi' 2026-03-20 16:26:25.794717 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.794722 | controller | - ' ' 2026-03-20 16:26:25.794726 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.794730 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.794734 | controller | | base64 -d)' 2026-03-20 16:26:25.794738 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.794743 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.794747 | controller | - ' exit 1' 2026-03-20 16:26:25.794751 | controller | - ' fi' 2026-03-20 16:26:25.794755 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.794760 | controller | - ' ' 2026-03-20 16:26:25.794768 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.794774 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.794779 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.794784 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.794789 | controller | --timeout=300s' 2026-03-20 16:26:25.794795 | controller | - ' ' 2026-03-20 16:26:25.794800 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.794804 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.794809 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.794813 | controller | - ' ' 2026-03-20 16:26:25.794817 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.794821 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.794825 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.794829 | controller | - ' ' 2026-03-20 16:26:25.794834 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.794838 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.794842 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.794846 | controller | available"' 2026-03-20 16:26:25.794850 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.794855 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.794859 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.794863 | controller | - ' exit 1' 2026-03-20 16:26:25.794867 | controller | - ' fi' 2026-03-20 16:26:25.794871 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.794875 | controller | - ' ' 2026-03-20 16:26:25.794880 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.794886 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.794892 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.794898 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.794903 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.794909 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.794915 | controller | -- \' 2026-03-20 16:26:25.794920 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.794926 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.794931 | controller | - ' logger.go:42: 16:15:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.794937 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.794943 | controller | - ' logger.go:42: 16:15:13 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.794958 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.794965 | controller | found' 2026-03-20 16:26:25.794970 | controller | - ' logger.go:42: 16:15:14 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.794976 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.794982 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.794988 | controller | - ' ' 2026-03-20 16:26:25.794993 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.794999 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.795004 | controller | - ' ' 2026-03-20 16:26:25.795013 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.795018 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.795022 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.795027 | controller | - ' exit 1' 2026-03-20 16:26:25.795031 | controller | - ' fi' 2026-03-20 16:26:25.795036 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.795048 | controller | - ' ' 2026-03-20 16:26:25.795064 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.795070 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.795076 | controller | | base64 -d)' 2026-03-20 16:26:25.795082 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.795088 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.795094 | controller | - ' exit 1' 2026-03-20 16:26:25.795100 | controller | - ' fi' 2026-03-20 16:26:25.795105 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.795110 | controller | - ' ' 2026-03-20 16:26:25.795114 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.795118 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.795123 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.795129 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.795134 | controller | --timeout=300s' 2026-03-20 16:26:25.795140 | controller | - ' ' 2026-03-20 16:26:25.795145 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.795149 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.795158 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.795162 | controller | - ' ' 2026-03-20 16:26:25.795167 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.795171 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.795175 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.795179 | controller | - ' ' 2026-03-20 16:26:25.795183 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.795188 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.795192 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.795196 | controller | available"' 2026-03-20 16:26:25.795200 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.795204 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.795208 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.795212 | controller | - ' exit 1' 2026-03-20 16:26:25.795217 | controller | - ' fi' 2026-03-20 16:26:25.795221 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.795225 | controller | - ' ' 2026-03-20 16:26:25.795230 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.795236 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.795242 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.795248 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.795254 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.795259 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.795265 | controller | -- \' 2026-03-20 16:26:25.795271 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.795276 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.795282 | controller | - ' logger.go:42: 16:15:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.795288 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.795292 | controller | - ' logger.go:42: 16:15:14 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.795296 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.795301 | controller | found' 2026-03-20 16:26:25.795317 | controller | - ' logger.go:42: 16:15:15 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.795323 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.795327 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.795331 | controller | - ' ' 2026-03-20 16:26:25.795336 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.795340 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.795344 | controller | - ' ' 2026-03-20 16:26:25.795348 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.795353 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.795357 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.795364 | controller | - ' exit 1' 2026-03-20 16:26:25.795368 | controller | - ' fi' 2026-03-20 16:26:25.795372 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.795377 | controller | - ' ' 2026-03-20 16:26:25.795381 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.795385 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.795389 | controller | | base64 -d)' 2026-03-20 16:26:25.795394 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.795398 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.795404 | controller | - ' exit 1' 2026-03-20 16:26:25.795414 | controller | - ' fi' 2026-03-20 16:26:25.795427 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.795433 | controller | - ' ' 2026-03-20 16:26:25.795439 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.795445 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.795451 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.795457 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.795463 | controller | --timeout=300s' 2026-03-20 16:26:25.795469 | controller | - ' ' 2026-03-20 16:26:25.795475 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.795481 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.795485 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.795489 | controller | - ' ' 2026-03-20 16:26:25.795494 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.795498 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.795502 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.795508 | controller | - ' ' 2026-03-20 16:26:25.795517 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.795531 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.795537 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.795542 | controller | available"' 2026-03-20 16:26:25.795548 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.795554 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.795560 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.795565 | controller | - ' exit 1' 2026-03-20 16:26:25.795571 | controller | - ' fi' 2026-03-20 16:26:25.795577 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.795582 | controller | - ' ' 2026-03-20 16:26:25.795588 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.795594 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.795600 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.795605 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.795611 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.795620 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.795630 | controller | -- \' 2026-03-20 16:26:25.795636 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.795641 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.795647 | controller | - ' logger.go:42: 16:15:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.795653 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.795658 | controller | - ' logger.go:42: 16:15:15 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.795664 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.795670 | controller | found' 2026-03-20 16:26:25.795690 | controller | - ' logger.go:42: 16:15:16 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.795741 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.795747 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.795753 | controller | - ' ' 2026-03-20 16:26:25.795758 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.795763 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.795769 | controller | - ' ' 2026-03-20 16:26:25.795775 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.795780 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.795786 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.795792 | controller | - ' exit 1' 2026-03-20 16:26:25.795798 | controller | - ' fi' 2026-03-20 16:26:25.795803 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.795813 | controller | - ' ' 2026-03-20 16:26:25.795819 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.795825 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.795831 | controller | | base64 -d)' 2026-03-20 16:26:25.795836 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.795842 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.795847 | controller | - ' exit 1' 2026-03-20 16:26:25.795853 | controller | - ' fi' 2026-03-20 16:26:25.795858 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.795864 | controller | - ' ' 2026-03-20 16:26:25.795870 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.795875 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.795881 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.795886 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.795892 | controller | --timeout=300s' 2026-03-20 16:26:25.795898 | controller | - ' ' 2026-03-20 16:26:25.795904 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.795910 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.795916 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.795921 | controller | - ' ' 2026-03-20 16:26:25.795926 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.795930 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.795935 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.795942 | controller | - ' ' 2026-03-20 16:26:25.795946 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.795950 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.795954 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.795958 | controller | available"' 2026-03-20 16:26:25.795963 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.795967 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.795971 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.795975 | controller | - ' exit 1' 2026-03-20 16:26:25.795979 | controller | - ' fi' 2026-03-20 16:26:25.795983 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.795988 | controller | - ' ' 2026-03-20 16:26:25.795992 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.795996 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.796000 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.796004 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.796009 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.796013 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.796017 | controller | -- \' 2026-03-20 16:26:25.796021 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.796025 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.796030 | controller | - ' logger.go:42: 16:15:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.796034 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.796038 | controller | - ' logger.go:42: 16:15:17 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.796042 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.796046 | controller | found' 2026-03-20 16:26:25.796051 | controller | - ' logger.go:42: 16:15:18 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.796055 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.796074 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.796090 | controller | - ' ' 2026-03-20 16:26:25.796097 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.796101 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.796105 | controller | - ' ' 2026-03-20 16:26:25.796110 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.796114 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.796118 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.796122 | controller | - ' exit 1' 2026-03-20 16:26:25.796126 | controller | - ' fi' 2026-03-20 16:26:25.796136 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.796149 | controller | - ' ' 2026-03-20 16:26:25.796160 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.796167 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.796173 | controller | | base64 -d)' 2026-03-20 16:26:25.796179 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.796189 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.796193 | controller | - ' exit 1' 2026-03-20 16:26:25.796198 | controller | - ' fi' 2026-03-20 16:26:25.796202 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.796206 | controller | - ' ' 2026-03-20 16:26:25.796210 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.796215 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.796219 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.796223 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.796227 | controller | --timeout=300s' 2026-03-20 16:26:25.796231 | controller | - ' ' 2026-03-20 16:26:25.796235 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.796240 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.796244 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.796248 | controller | - ' ' 2026-03-20 16:26:25.796252 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.796256 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.796261 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.796265 | controller | - ' ' 2026-03-20 16:26:25.796269 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.796273 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.796280 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.796285 | controller | available"' 2026-03-20 16:26:25.796289 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.796293 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.796297 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.796302 | controller | - ' exit 1' 2026-03-20 16:26:25.796306 | controller | - ' fi' 2026-03-20 16:26:25.796310 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.796314 | controller | - ' ' 2026-03-20 16:26:25.796318 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.796322 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.796327 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.796331 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.796335 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.796339 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.796344 | controller | -- \' 2026-03-20 16:26:25.796348 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.796352 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.796356 | controller | - ' logger.go:42: 16:15:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.796360 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.796367 | controller | - ' logger.go:42: 16:15:18 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.796372 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.796376 | controller | found' 2026-03-20 16:26:25.796380 | controller | - ' logger.go:42: 16:15:19 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.796384 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.796388 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.796393 | controller | - ' ' 2026-03-20 16:26:25.796408 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.796413 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.796418 | controller | - ' ' 2026-03-20 16:26:25.796422 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.796426 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.796430 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.796435 | controller | - ' exit 1' 2026-03-20 16:26:25.796439 | controller | - ' fi' 2026-03-20 16:26:25.796443 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.796447 | controller | - ' ' 2026-03-20 16:26:25.796451 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.796456 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.796460 | controller | | base64 -d)' 2026-03-20 16:26:25.796464 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.796468 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.796472 | controller | - ' exit 1' 2026-03-20 16:26:25.796476 | controller | - ' fi' 2026-03-20 16:26:25.796481 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.796485 | controller | - ' ' 2026-03-20 16:26:25.796489 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.796493 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.796499 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.796504 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.796508 | controller | --timeout=300s' 2026-03-20 16:26:25.796512 | controller | - ' ' 2026-03-20 16:26:25.796516 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.796521 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.796525 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.796529 | controller | - ' ' 2026-03-20 16:26:25.796533 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.796537 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.796542 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.796546 | controller | - ' ' 2026-03-20 16:26:25.796550 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.796554 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.796558 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.796563 | controller | available"' 2026-03-20 16:26:25.796567 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.796574 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.796579 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.796583 | controller | - ' exit 1' 2026-03-20 16:26:25.796587 | controller | - ' fi' 2026-03-20 16:26:25.796591 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.796595 | controller | - ' ' 2026-03-20 16:26:25.796600 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.796604 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.796608 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.796612 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.796616 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.796620 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.796625 | controller | -- \' 2026-03-20 16:26:25.796629 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.796633 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.796637 | controller | - ' logger.go:42: 16:15:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.796641 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.796645 | controller | - ' logger.go:42: 16:15:19 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.796650 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.796654 | controller | found' 2026-03-20 16:26:25.796658 | controller | - ' logger.go:42: 16:15:20 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.796662 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.796666 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.796671 | controller | - ' ' 2026-03-20 16:26:25.796684 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.796688 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.796717 | controller | - ' ' 2026-03-20 16:26:25.796733 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.796738 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.796742 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.796747 | controller | - ' exit 1' 2026-03-20 16:26:25.796751 | controller | - ' fi' 2026-03-20 16:26:25.796755 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.796759 | controller | - ' ' 2026-03-20 16:26:25.796763 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.796767 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.796772 | controller | | base64 -d)' 2026-03-20 16:26:25.796776 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.796780 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.796784 | controller | - ' exit 1' 2026-03-20 16:26:25.796788 | controller | - ' fi' 2026-03-20 16:26:25.796792 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.796797 | controller | - ' ' 2026-03-20 16:26:25.796801 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.796805 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.796813 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.796817 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.796821 | controller | --timeout=300s' 2026-03-20 16:26:25.796825 | controller | - ' ' 2026-03-20 16:26:25.796829 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.796833 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.796838 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.796842 | controller | - ' ' 2026-03-20 16:26:25.796846 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.796850 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.796858 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.796862 | controller | - ' ' 2026-03-20 16:26:25.796866 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.796871 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.796875 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.796879 | controller | available"' 2026-03-20 16:26:25.796883 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.796887 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.796891 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.796896 | controller | - ' exit 1' 2026-03-20 16:26:25.796900 | controller | - ' fi' 2026-03-20 16:26:25.796904 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.796908 | controller | - ' ' 2026-03-20 16:26:25.796912 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.796916 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.796920 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.796925 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.796929 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.796933 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.796937 | controller | -- \' 2026-03-20 16:26:25.796941 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.796945 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.796949 | controller | - ' logger.go:42: 16:15:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.796957 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.796961 | controller | - ' logger.go:42: 16:15:20 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.796965 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.796969 | controller | found' 2026-03-20 16:26:25.796974 | controller | - ' logger.go:42: 16:15:21 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.796978 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.796982 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.796986 | controller | - ' ' 2026-03-20 16:26:25.796992 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.797007 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.797012 | controller | - ' ' 2026-03-20 16:26:25.797017 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.797021 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.797025 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.797029 | controller | - ' exit 1' 2026-03-20 16:26:25.797033 | controller | - ' fi' 2026-03-20 16:26:25.797038 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.797042 | controller | - ' ' 2026-03-20 16:26:25.797046 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.797050 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.797054 | controller | | base64 -d)' 2026-03-20 16:26:25.797058 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.797063 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.797067 | controller | - ' exit 1' 2026-03-20 16:26:25.797071 | controller | - ' fi' 2026-03-20 16:26:25.797075 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.797079 | controller | - ' ' 2026-03-20 16:26:25.797084 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.797088 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.797092 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.797098 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.797103 | controller | --timeout=300s' 2026-03-20 16:26:25.797107 | controller | - ' ' 2026-03-20 16:26:25.797111 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.797115 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.797119 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.797123 | controller | - ' ' 2026-03-20 16:26:25.797128 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.797132 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.797136 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.797140 | controller | - ' ' 2026-03-20 16:26:25.797144 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.797148 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.797152 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.797157 | controller | available"' 2026-03-20 16:26:25.797161 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.797165 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.797169 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.797173 | controller | - ' exit 1' 2026-03-20 16:26:25.797177 | controller | - ' fi' 2026-03-20 16:26:25.797183 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.797189 | controller | - ' ' 2026-03-20 16:26:25.797193 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.797199 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.797203 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.797207 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.797211 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.797215 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.797221 | controller | -- \' 2026-03-20 16:26:25.797227 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.797243 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.797251 | controller | - ' logger.go:42: 16:15:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.797257 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.797263 | controller | - ' logger.go:42: 16:15:21 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.797269 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.797274 | controller | found' 2026-03-20 16:26:25.797280 | controller | - ' logger.go:42: 16:15:22 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.797285 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.797291 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.797296 | controller | - ' ' 2026-03-20 16:26:25.797302 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.797320 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.797326 | controller | - ' ' 2026-03-20 16:26:25.797333 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.797338 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.797344 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.797350 | controller | - ' exit 1' 2026-03-20 16:26:25.797356 | controller | - ' fi' 2026-03-20 16:26:25.797361 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.797367 | controller | - ' ' 2026-03-20 16:26:25.797372 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.797378 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.797383 | controller | | base64 -d)' 2026-03-20 16:26:25.797389 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.797394 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.797400 | controller | - ' exit 1' 2026-03-20 16:26:25.797405 | controller | - ' fi' 2026-03-20 16:26:25.797411 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.797417 | controller | - ' ' 2026-03-20 16:26:25.797423 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.797428 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.797434 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.797439 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.797445 | controller | --timeout=300s' 2026-03-20 16:26:25.797451 | controller | - ' ' 2026-03-20 16:26:25.797456 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.797462 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.797473 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.797479 | controller | - ' ' 2026-03-20 16:26:25.797485 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.797491 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.797497 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.797503 | controller | - ' ' 2026-03-20 16:26:25.797509 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.797514 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.797520 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.797526 | controller | available"' 2026-03-20 16:26:25.797532 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.797537 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.797543 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.797548 | controller | - ' exit 1' 2026-03-20 16:26:25.797554 | controller | - ' fi' 2026-03-20 16:26:25.797560 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.797566 | controller | - ' ' 2026-03-20 16:26:25.797572 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.797577 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.797583 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.797589 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.797595 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.797600 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.797606 | controller | -- \' 2026-03-20 16:26:25.797611 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.797617 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.797622 | controller | - ' logger.go:42: 16:15:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.797628 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.797634 | controller | - ' logger.go:42: 16:15:22 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.797639 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.797645 | controller | found' 2026-03-20 16:26:25.797650 | controller | - ' logger.go:42: 16:15:23 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.797656 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.797662 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.797668 | controller | - ' ' 2026-03-20 16:26:25.797673 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.797679 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.797685 | controller | - ' ' 2026-03-20 16:26:25.797723 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.797735 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.797741 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.797747 | controller | - ' exit 1' 2026-03-20 16:26:25.797756 | controller | - ' fi' 2026-03-20 16:26:25.797762 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.797767 | controller | - ' ' 2026-03-20 16:26:25.797773 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.797778 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.797784 | controller | | base64 -d)' 2026-03-20 16:26:25.797790 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.797795 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.797801 | controller | - ' exit 1' 2026-03-20 16:26:25.797807 | controller | - ' fi' 2026-03-20 16:26:25.797812 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.797823 | controller | - ' ' 2026-03-20 16:26:25.797829 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.797834 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.797840 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.797846 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.797851 | controller | --timeout=300s' 2026-03-20 16:26:25.797857 | controller | - ' ' 2026-03-20 16:26:25.797863 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.797869 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.797875 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.797881 | controller | - ' ' 2026-03-20 16:26:25.797886 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.797892 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.797898 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.797904 | controller | - ' ' 2026-03-20 16:26:25.797913 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.797920 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.797926 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.797932 | controller | available"' 2026-03-20 16:26:25.797939 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.797945 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.797951 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.797957 | controller | - ' exit 1' 2026-03-20 16:26:25.797963 | controller | - ' fi' 2026-03-20 16:26:25.797968 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.797973 | controller | - ' ' 2026-03-20 16:26:25.797977 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.797981 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.797986 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.797990 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.797994 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.797999 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.798006 | controller | -- \' 2026-03-20 16:26:25.798010 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.798014 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.798021 | controller | - ' logger.go:42: 16:15:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.798025 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.798030 | controller | - ' logger.go:42: 16:15:24 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.798034 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.798038 | controller | found' 2026-03-20 16:26:25.798043 | controller | - ' logger.go:42: 16:15:25 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.798047 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.798051 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.798056 | controller | - ' ' 2026-03-20 16:26:25.798060 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.798064 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.798069 | controller | - ' ' 2026-03-20 16:26:25.798084 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.798090 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.798095 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.798099 | controller | - ' exit 1' 2026-03-20 16:26:25.798103 | controller | - ' fi' 2026-03-20 16:26:25.798108 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.798112 | controller | - ' ' 2026-03-20 16:26:25.798116 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.798121 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.798125 | controller | | base64 -d)' 2026-03-20 16:26:25.798129 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.798133 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.798140 | controller | - ' exit 1' 2026-03-20 16:26:25.798144 | controller | - ' fi' 2026-03-20 16:26:25.798149 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.798153 | controller | - ' ' 2026-03-20 16:26:25.798157 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.798161 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.798166 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.798170 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.798174 | controller | --timeout=300s' 2026-03-20 16:26:25.798179 | controller | - ' ' 2026-03-20 16:26:25.798183 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.798187 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.798192 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.798196 | controller | - ' ' 2026-03-20 16:26:25.798200 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.798204 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.798209 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.798215 | controller | - ' ' 2026-03-20 16:26:25.798219 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.798223 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.798228 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.798232 | controller | available"' 2026-03-20 16:26:25.798236 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.798240 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.798245 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.798249 | controller | - ' exit 1' 2026-03-20 16:26:25.798255 | controller | - ' fi' 2026-03-20 16:26:25.798259 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.798264 | controller | - ' ' 2026-03-20 16:26:25.798268 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.798272 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.798277 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.798281 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.798285 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.798289 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.798294 | controller | -- \' 2026-03-20 16:26:25.798298 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.798302 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.798307 | controller | - ' logger.go:42: 16:15:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.798312 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.798317 | controller | - ' logger.go:42: 16:15:25 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.798322 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.798327 | controller | found' 2026-03-20 16:26:25.798332 | controller | - ' logger.go:42: 16:15:26 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.798337 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.798341 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.798346 | controller | - ' ' 2026-03-20 16:26:25.798350 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.798355 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.798359 | controller | - ' ' 2026-03-20 16:26:25.798363 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.798368 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.798380 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.798385 | controller | - ' exit 1' 2026-03-20 16:26:25.798391 | controller | - ' fi' 2026-03-20 16:26:25.798399 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.798405 | controller | - ' ' 2026-03-20 16:26:25.798411 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.798417 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.798421 | controller | | base64 -d)' 2026-03-20 16:26:25.798426 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.798430 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.798438 | controller | - ' exit 1' 2026-03-20 16:26:25.798442 | controller | - ' fi' 2026-03-20 16:26:25.798447 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.798451 | controller | - ' ' 2026-03-20 16:26:25.798455 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.798460 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.798464 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.798468 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.798473 | controller | --timeout=300s' 2026-03-20 16:26:25.798477 | controller | - ' ' 2026-03-20 16:26:25.798484 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.798489 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.798493 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.798498 | controller | - ' ' 2026-03-20 16:26:25.798502 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.798507 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.798513 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.798519 | controller | - ' ' 2026-03-20 16:26:25.798525 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.798532 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.798539 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.798543 | controller | available"' 2026-03-20 16:26:25.798548 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.798552 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.798556 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.798560 | controller | - ' exit 1' 2026-03-20 16:26:25.798565 | controller | - ' fi' 2026-03-20 16:26:25.798569 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.798574 | controller | - ' ' 2026-03-20 16:26:25.798580 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.798586 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.798592 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.798598 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.798604 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.798611 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.798617 | controller | -- \' 2026-03-20 16:26:25.798623 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.798629 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.798635 | controller | - ' logger.go:42: 16:15:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.798641 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.798647 | controller | - ' logger.go:42: 16:15:26 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.798655 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.798661 | controller | found' 2026-03-20 16:26:25.798667 | controller | - ' logger.go:42: 16:15:27 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.798673 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.798679 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.798685 | controller | - ' ' 2026-03-20 16:26:25.798691 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.798718 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.798723 | controller | - ' ' 2026-03-20 16:26:25.798727 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.798732 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.798746 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.798750 | controller | - ' exit 1' 2026-03-20 16:26:25.798755 | controller | - ' fi' 2026-03-20 16:26:25.798759 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.798763 | controller | - ' ' 2026-03-20 16:26:25.798767 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.798772 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.798776 | controller | | base64 -d)' 2026-03-20 16:26:25.798782 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.798788 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.798794 | controller | - ' exit 1' 2026-03-20 16:26:25.798800 | controller | - ' fi' 2026-03-20 16:26:25.798805 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.798811 | controller | - ' ' 2026-03-20 16:26:25.798817 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.798821 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.798825 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.798829 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.798834 | controller | --timeout=300s' 2026-03-20 16:26:25.798838 | controller | - ' ' 2026-03-20 16:26:25.798842 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.798846 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.798851 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.798855 | controller | - ' ' 2026-03-20 16:26:25.798859 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.798863 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.798867 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.798871 | controller | - ' ' 2026-03-20 16:26:25.798876 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.798880 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.798884 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.798889 | controller | available"' 2026-03-20 16:26:25.798893 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.798897 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.798904 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.798908 | controller | - ' exit 1' 2026-03-20 16:26:25.798913 | controller | - ' fi' 2026-03-20 16:26:25.798917 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.798921 | controller | - ' ' 2026-03-20 16:26:25.798926 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.798930 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.798934 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.798938 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.798942 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.798947 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.798951 | controller | -- \' 2026-03-20 16:26:25.798955 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.798959 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.798964 | controller | - ' logger.go:42: 16:15:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.798968 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.798972 | controller | - ' logger.go:42: 16:15:27 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.798976 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.798981 | controller | found' 2026-03-20 16:26:25.798985 | controller | - ' logger.go:42: 16:15:28 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.798989 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.798993 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.798998 | controller | - ' ' 2026-03-20 16:26:25.799002 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.799006 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.799010 | controller | - ' ' 2026-03-20 16:26:25.799014 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.799019 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.799023 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.799027 | controller | - ' exit 1' 2026-03-20 16:26:25.799039 | controller | - ' fi' 2026-03-20 16:26:25.799044 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.799062 | controller | - ' ' 2026-03-20 16:26:25.799066 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.799071 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.799076 | controller | | base64 -d)' 2026-03-20 16:26:25.799082 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.799088 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.799094 | controller | - ' exit 1' 2026-03-20 16:26:25.799100 | controller | - ' fi' 2026-03-20 16:26:25.799106 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.799111 | controller | - ' ' 2026-03-20 16:26:25.799117 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.799123 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.799129 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.799138 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.799144 | controller | --timeout=300s' 2026-03-20 16:26:25.799150 | controller | - ' ' 2026-03-20 16:26:25.799156 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.799161 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.799167 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.799173 | controller | - ' ' 2026-03-20 16:26:25.799179 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.799185 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.799190 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.799196 | controller | - ' ' 2026-03-20 16:26:25.799202 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.799212 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.799217 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.799223 | controller | available"' 2026-03-20 16:26:25.799228 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.799234 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.799240 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.799245 | controller | - ' exit 1' 2026-03-20 16:26:25.799251 | controller | - ' fi' 2026-03-20 16:26:25.799257 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.799280 | controller | - ' ' 2026-03-20 16:26:25.799286 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.799292 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.799297 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.799303 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.799309 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.799314 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.799321 | controller | -- \' 2026-03-20 16:26:25.799326 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.799332 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.799338 | controller | - ' logger.go:42: 16:15:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.799344 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.799350 | controller | - ' logger.go:42: 16:15:28 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.799356 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.799362 | controller | found' 2026-03-20 16:26:25.799368 | controller | - ' logger.go:42: 16:15:29 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.799374 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.799380 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.799385 | controller | - ' ' 2026-03-20 16:26:25.799391 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.799401 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.799407 | controller | - ' ' 2026-03-20 16:26:25.799413 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.799419 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.799425 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.799435 | controller | - ' exit 1' 2026-03-20 16:26:25.799441 | controller | - ' fi' 2026-03-20 16:26:25.799459 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.799466 | controller | - ' ' 2026-03-20 16:26:25.799471 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.799475 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.799479 | controller | | base64 -d)' 2026-03-20 16:26:25.799484 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.799488 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.799492 | controller | - ' exit 1' 2026-03-20 16:26:25.799498 | controller | - ' fi' 2026-03-20 16:26:25.799504 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.799510 | controller | - ' ' 2026-03-20 16:26:25.799516 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.799522 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.799528 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.799534 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.799539 | controller | --timeout=300s' 2026-03-20 16:26:25.799545 | controller | - ' ' 2026-03-20 16:26:25.799551 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.799557 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.799563 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.799569 | controller | - ' ' 2026-03-20 16:26:25.799574 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.799580 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.799586 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.799592 | controller | - ' ' 2026-03-20 16:26:25.799598 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.799604 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.799609 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.799615 | controller | available"' 2026-03-20 16:26:25.799621 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.799640 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.799646 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.799651 | controller | - ' exit 1' 2026-03-20 16:26:25.799655 | controller | - ' fi' 2026-03-20 16:26:25.799659 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.799664 | controller | - ' ' 2026-03-20 16:26:25.799668 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.799672 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.799679 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.799683 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.799687 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.799730 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.799742 | controller | -- \' 2026-03-20 16:26:25.799749 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.799755 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.799761 | controller | - ' logger.go:42: 16:15:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.799767 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.799772 | controller | - ' logger.go:42: 16:15:29 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.799778 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.799784 | controller | found' 2026-03-20 16:26:25.799789 | controller | - ' logger.go:42: 16:15:30 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.799794 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.799798 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.799803 | controller | - ' ' 2026-03-20 16:26:25.799809 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.799815 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.799821 | controller | - ' ' 2026-03-20 16:26:25.799827 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.799832 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.799838 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.799844 | controller | - ' exit 1' 2026-03-20 16:26:25.799850 | controller | - ' fi' 2026-03-20 16:26:25.799859 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.799865 | controller | - ' ' 2026-03-20 16:26:25.799883 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.799889 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.799895 | controller | | base64 -d)' 2026-03-20 16:26:25.799904 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.799911 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.799916 | controller | - ' exit 1' 2026-03-20 16:26:25.799922 | controller | - ' fi' 2026-03-20 16:26:25.799928 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.799934 | controller | - ' ' 2026-03-20 16:26:25.799940 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.799962 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.799968 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.799974 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.799980 | controller | --timeout=300s' 2026-03-20 16:26:25.799986 | controller | - ' ' 2026-03-20 16:26:25.799991 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.799997 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.800003 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.800012 | controller | - ' ' 2026-03-20 16:26:25.800018 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.800022 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.800026 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.800031 | controller | - ' ' 2026-03-20 16:26:25.800035 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.800039 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.800043 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.800047 | controller | available"' 2026-03-20 16:26:25.800052 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.800056 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.800060 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.800064 | controller | - ' exit 1' 2026-03-20 16:26:25.800068 | controller | - ' fi' 2026-03-20 16:26:25.800073 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.800077 | controller | - ' ' 2026-03-20 16:26:25.800085 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.800089 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.800093 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.800098 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.800102 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.800106 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.800111 | controller | -- \' 2026-03-20 16:26:25.800116 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.800120 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.800124 | controller | - ' logger.go:42: 16:15:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.800128 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.800141 | controller | - ' logger.go:42: 16:15:30 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.800146 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.800150 | controller | found' 2026-03-20 16:26:25.800154 | controller | - ' logger.go:42: 16:15:31 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.800158 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.800162 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.800167 | controller | - ' ' 2026-03-20 16:26:25.800171 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.800175 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.800179 | controller | - ' ' 2026-03-20 16:26:25.800183 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.800187 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.800192 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.800196 | controller | - ' exit 1' 2026-03-20 16:26:25.800200 | controller | - ' fi' 2026-03-20 16:26:25.800207 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.800212 | controller | - ' ' 2026-03-20 16:26:25.800216 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.800229 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.800234 | controller | | base64 -d)' 2026-03-20 16:26:25.800239 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.800243 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.800247 | controller | - ' exit 1' 2026-03-20 16:26:25.800251 | controller | - ' fi' 2026-03-20 16:26:25.800256 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.800260 | controller | - ' ' 2026-03-20 16:26:25.800264 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.800268 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.800272 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.800277 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.800281 | controller | --timeout=300s' 2026-03-20 16:26:25.800285 | controller | - ' ' 2026-03-20 16:26:25.800289 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.800294 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.800298 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.800302 | controller | - ' ' 2026-03-20 16:26:25.800306 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.800310 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.800315 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.800319 | controller | - ' ' 2026-03-20 16:26:25.800323 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.800327 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.800331 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.800336 | controller | available"' 2026-03-20 16:26:25.800340 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.800344 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.800348 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.800353 | controller | - ' exit 1' 2026-03-20 16:26:25.800357 | controller | - ' fi' 2026-03-20 16:26:25.800361 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.800365 | controller | - ' ' 2026-03-20 16:26:25.800369 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.800374 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.800378 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.800383 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.800387 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.800391 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.800398 | controller | -- \' 2026-03-20 16:26:25.800402 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.800406 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.800411 | controller | - ' logger.go:42: 16:15:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.800415 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.800419 | controller | - ' logger.go:42: 16:15:32 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.800423 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.800428 | controller | found' 2026-03-20 16:26:25.800432 | controller | - ' logger.go:42: 16:15:33 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.800436 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.800440 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.800444 | controller | - ' ' 2026-03-20 16:26:25.800449 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.800453 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.800457 | controller | - ' ' 2026-03-20 16:26:25.800461 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.800466 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.800470 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.800474 | controller | - ' exit 1' 2026-03-20 16:26:25.800478 | controller | - ' fi' 2026-03-20 16:26:25.800482 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.800487 | controller | - ' ' 2026-03-20 16:26:25.800491 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.800502 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.800506 | controller | | base64 -d)' 2026-03-20 16:26:25.800510 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.800515 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.800520 | controller | - ' exit 1' 2026-03-20 16:26:25.800525 | controller | - ' fi' 2026-03-20 16:26:25.800531 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.800537 | controller | - ' ' 2026-03-20 16:26:25.800542 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.800548 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.800554 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.800560 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.800566 | controller | --timeout=300s' 2026-03-20 16:26:25.800571 | controller | - ' ' 2026-03-20 16:26:25.800576 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.800582 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.800591 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.800596 | controller | - ' ' 2026-03-20 16:26:25.800602 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.800607 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.800613 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.800619 | controller | - ' ' 2026-03-20 16:26:25.800629 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.800634 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.800639 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.800645 | controller | available"' 2026-03-20 16:26:25.800650 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.800656 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.800661 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.800667 | controller | - ' exit 1' 2026-03-20 16:26:25.800672 | controller | - ' fi' 2026-03-20 16:26:25.800677 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.800683 | controller | - ' ' 2026-03-20 16:26:25.800688 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.800711 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.800721 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.800730 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.800736 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.800742 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.800749 | controller | -- \' 2026-03-20 16:26:25.800755 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.800761 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.800767 | controller | - ' logger.go:42: 16:15:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.800773 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.800779 | controller | - ' logger.go:42: 16:15:33 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.800784 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.800790 | controller | found' 2026-03-20 16:26:25.800796 | controller | - ' logger.go:42: 16:15:34 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.800802 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.800808 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.800814 | controller | - ' ' 2026-03-20 16:26:25.800820 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.800826 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.800832 | controller | - ' ' 2026-03-20 16:26:25.800838 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.800844 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.800850 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.800856 | controller | - ' exit 1' 2026-03-20 16:26:25.800862 | controller | - ' fi' 2026-03-20 16:26:25.800868 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.800874 | controller | - ' ' 2026-03-20 16:26:25.800880 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.800886 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.800903 | controller | | base64 -d)' 2026-03-20 16:26:25.800909 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.800915 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.800924 | controller | - ' exit 1' 2026-03-20 16:26:25.800931 | controller | - ' fi' 2026-03-20 16:26:25.800936 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.800942 | controller | - ' ' 2026-03-20 16:26:25.800948 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.800954 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.800960 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.800966 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.800972 | controller | --timeout=300s' 2026-03-20 16:26:25.800978 | controller | - ' ' 2026-03-20 16:26:25.800984 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.800989 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.800995 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.801001 | controller | - ' ' 2026-03-20 16:26:25.801007 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.801013 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.801019 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.801025 | controller | - ' ' 2026-03-20 16:26:25.801030 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.801036 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.801045 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.801050 | controller | available"' 2026-03-20 16:26:25.801056 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.801062 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.801068 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.801074 | controller | - ' exit 1' 2026-03-20 16:26:25.801080 | controller | - ' fi' 2026-03-20 16:26:25.801086 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.801093 | controller | - ' ' 2026-03-20 16:26:25.801099 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.801105 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.801110 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.801115 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.801119 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.801123 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.801127 | controller | -- \' 2026-03-20 16:26:25.801132 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.801136 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.801140 | controller | - ' logger.go:42: 16:15:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.801144 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.801148 | controller | - ' logger.go:42: 16:15:34 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.801155 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.801159 | controller | found' 2026-03-20 16:26:25.801163 | controller | - ' logger.go:42: 16:15:35 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.801168 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.801172 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.801176 | controller | - ' ' 2026-03-20 16:26:25.801183 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.801187 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.801191 | controller | - ' ' 2026-03-20 16:26:25.801195 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.801199 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.801204 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.801208 | controller | - ' exit 1' 2026-03-20 16:26:25.801212 | controller | - ' fi' 2026-03-20 16:26:25.801216 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.801221 | controller | - ' ' 2026-03-20 16:26:25.801226 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.801231 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.801235 | controller | | base64 -d)' 2026-03-20 16:26:25.801249 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.801253 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.801257 | controller | - ' exit 1' 2026-03-20 16:26:25.801262 | controller | - ' fi' 2026-03-20 16:26:25.801266 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.801270 | controller | - ' ' 2026-03-20 16:26:25.801274 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.801278 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.801282 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.801287 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.801291 | controller | --timeout=300s' 2026-03-20 16:26:25.801295 | controller | - ' ' 2026-03-20 16:26:25.801299 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.801303 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.801308 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.801312 | controller | - ' ' 2026-03-20 16:26:25.801316 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.801320 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.801324 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.801329 | controller | - ' ' 2026-03-20 16:26:25.801333 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.801337 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.801341 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.801345 | controller | available"' 2026-03-20 16:26:25.801349 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.801353 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.801360 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.801364 | controller | - ' exit 1' 2026-03-20 16:26:25.801369 | controller | - ' fi' 2026-03-20 16:26:25.801373 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.801377 | controller | - ' ' 2026-03-20 16:26:25.801381 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.801386 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.801390 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.801394 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.801398 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.801403 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.801407 | controller | -- \' 2026-03-20 16:26:25.801411 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.801417 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.801422 | controller | - ' logger.go:42: 16:15:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.801426 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.801430 | controller | - ' logger.go:42: 16:15:35 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.801434 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.801439 | controller | found' 2026-03-20 16:26:25.801443 | controller | - ' logger.go:42: 16:15:36 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.801447 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.801451 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.801456 | controller | - ' ' 2026-03-20 16:26:25.801460 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.801464 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.801468 | controller | - ' ' 2026-03-20 16:26:25.801472 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.801476 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.801481 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.801485 | controller | - ' exit 1' 2026-03-20 16:26:25.801489 | controller | - ' fi' 2026-03-20 16:26:25.801493 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.801498 | controller | - ' ' 2026-03-20 16:26:25.801502 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.801506 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.801510 | controller | | base64 -d)' 2026-03-20 16:26:25.801514 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.801525 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.801530 | controller | - ' exit 1' 2026-03-20 16:26:25.801534 | controller | - ' fi' 2026-03-20 16:26:25.801538 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.801542 | controller | - ' ' 2026-03-20 16:26:25.801547 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.801551 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.801555 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.801559 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.801565 | controller | --timeout=300s' 2026-03-20 16:26:25.801569 | controller | - ' ' 2026-03-20 16:26:25.801574 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.801578 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.801582 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.801586 | controller | - ' ' 2026-03-20 16:26:25.801591 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.801595 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.801599 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.801603 | controller | - ' ' 2026-03-20 16:26:25.801607 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.801612 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.801616 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.801620 | controller | available"' 2026-03-20 16:26:25.801624 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.801629 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.801633 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.801637 | controller | - ' exit 1' 2026-03-20 16:26:25.801641 | controller | - ' fi' 2026-03-20 16:26:25.801645 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.801649 | controller | - ' ' 2026-03-20 16:26:25.801654 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.801658 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.801662 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.801666 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.801672 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.801677 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.801681 | controller | -- \' 2026-03-20 16:26:25.801685 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.801689 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.801708 | controller | - ' logger.go:42: 16:15:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.801713 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.801717 | controller | - ' logger.go:42: 16:15:36 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.801721 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.801725 | controller | found' 2026-03-20 16:26:25.801729 | controller | - ' logger.go:42: 16:15:37 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.801733 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.801738 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.801742 | controller | - ' ' 2026-03-20 16:26:25.801746 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.801750 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.801758 | controller | - ' ' 2026-03-20 16:26:25.801762 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.801766 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.801771 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.801775 | controller | - ' exit 1' 2026-03-20 16:26:25.801779 | controller | - ' fi' 2026-03-20 16:26:25.801783 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.801787 | controller | - ' ' 2026-03-20 16:26:25.801792 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.801796 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.801800 | controller | | base64 -d)' 2026-03-20 16:26:25.801804 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.801815 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.801820 | controller | - ' exit 1' 2026-03-20 16:26:25.801824 | controller | - ' fi' 2026-03-20 16:26:25.801829 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.801833 | controller | - ' ' 2026-03-20 16:26:25.801837 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.801841 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.801846 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.801850 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.801854 | controller | --timeout=300s' 2026-03-20 16:26:25.801858 | controller | - ' ' 2026-03-20 16:26:25.801862 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.801867 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.801871 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.801875 | controller | - ' ' 2026-03-20 16:26:25.801879 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.801883 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.801888 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.801892 | controller | - ' ' 2026-03-20 16:26:25.801896 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.801900 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.801904 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.801908 | controller | available"' 2026-03-20 16:26:25.801913 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.801917 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.801921 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.801925 | controller | - ' exit 1' 2026-03-20 16:26:25.801929 | controller | - ' fi' 2026-03-20 16:26:25.801934 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.801938 | controller | - ' ' 2026-03-20 16:26:25.801942 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.801946 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.801950 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.801956 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.801961 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.801965 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.801969 | controller | -- \' 2026-03-20 16:26:25.801973 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.801978 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.801982 | controller | - ' logger.go:42: 16:15:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.801986 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.801992 | controller | - ' logger.go:42: 16:15:37 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.801996 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.802001 | controller | found' 2026-03-20 16:26:25.802005 | controller | - ' logger.go:42: 16:15:38 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.802009 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.802013 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.802017 | controller | - ' ' 2026-03-20 16:26:25.802021 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.802026 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.802030 | controller | - ' ' 2026-03-20 16:26:25.802034 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.802038 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.802042 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.802046 | controller | - ' exit 1' 2026-03-20 16:26:25.802051 | controller | - ' fi' 2026-03-20 16:26:25.802055 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.802059 | controller | - ' ' 2026-03-20 16:26:25.802063 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.802067 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.802072 | controller | | base64 -d)' 2026-03-20 16:26:25.802076 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.802080 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.802084 | controller | - ' exit 1' 2026-03-20 16:26:25.802088 | controller | - ' fi' 2026-03-20 16:26:25.802099 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.802104 | controller | - ' ' 2026-03-20 16:26:25.802108 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.802112 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.802116 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.802120 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.802125 | controller | --timeout=300s' 2026-03-20 16:26:25.802129 | controller | - ' ' 2026-03-20 16:26:25.802133 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.802137 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.802141 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.802147 | controller | - ' ' 2026-03-20 16:26:25.802152 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.802156 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.802160 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.802164 | controller | - ' ' 2026-03-20 16:26:25.802171 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.802175 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.802179 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.802183 | controller | available"' 2026-03-20 16:26:25.802188 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.802192 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.802196 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.802200 | controller | - ' exit 1' 2026-03-20 16:26:25.802205 | controller | - ' fi' 2026-03-20 16:26:25.802209 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.802213 | controller | - ' ' 2026-03-20 16:26:25.802218 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.802222 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.802226 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.802230 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.802234 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.802238 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.802243 | controller | -- \' 2026-03-20 16:26:25.802247 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.802251 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.802255 | controller | - ' logger.go:42: 16:15:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.802259 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.802264 | controller | - ' logger.go:42: 16:15:38 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.802268 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.802272 | controller | found' 2026-03-20 16:26:25.802276 | controller | - ' logger.go:42: 16:15:39 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.802281 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.802285 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.802289 | controller | - ' ' 2026-03-20 16:26:25.802293 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.802297 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.802301 | controller | - ' ' 2026-03-20 16:26:25.802306 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.802310 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.802314 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.802318 | controller | - ' exit 1' 2026-03-20 16:26:25.802323 | controller | - ' fi' 2026-03-20 16:26:25.802327 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.802333 | controller | - ' ' 2026-03-20 16:26:25.802337 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.802341 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.802345 | controller | | base64 -d)' 2026-03-20 16:26:25.802350 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.802354 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.802358 | controller | - ' exit 1' 2026-03-20 16:26:25.802362 | controller | - ' fi' 2026-03-20 16:26:25.802373 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.802389 | controller | - ' ' 2026-03-20 16:26:25.802394 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.802398 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.802402 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.802406 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.802410 | controller | --timeout=300s' 2026-03-20 16:26:25.802416 | controller | - ' ' 2026-03-20 16:26:25.802420 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.802425 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.802429 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.802433 | controller | - ' ' 2026-03-20 16:26:25.802437 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.802441 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.802446 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.802450 | controller | - ' ' 2026-03-20 16:26:25.802454 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.802458 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.802462 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.802467 | controller | available"' 2026-03-20 16:26:25.802471 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.802476 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.802481 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.802485 | controller | - ' exit 1' 2026-03-20 16:26:25.802489 | controller | - ' fi' 2026-03-20 16:26:25.802493 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.802497 | controller | - ' ' 2026-03-20 16:26:25.802502 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.802506 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.802510 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.802514 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.802518 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.802522 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.802527 | controller | -- \' 2026-03-20 16:26:25.802531 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.802537 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.802541 | controller | - ' logger.go:42: 16:15:39 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.802546 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.802557 | controller | - ' logger.go:42: 16:15:40 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.802562 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.802567 | controller | found' 2026-03-20 16:26:25.802572 | controller | - ' logger.go:42: 16:15:41 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.802576 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.802580 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.802584 | controller | - ' ' 2026-03-20 16:26:25.802589 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.802593 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.802597 | controller | - ' ' 2026-03-20 16:26:25.802601 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.802605 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.802610 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.802614 | controller | - ' exit 1' 2026-03-20 16:26:25.802618 | controller | - ' fi' 2026-03-20 16:26:25.802622 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.802626 | controller | - ' ' 2026-03-20 16:26:25.802631 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.802635 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.802639 | controller | | base64 -d)' 2026-03-20 16:26:25.802643 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.802647 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.802651 | controller | - ' exit 1' 2026-03-20 16:26:25.802656 | controller | - ' fi' 2026-03-20 16:26:25.802660 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.802664 | controller | - ' ' 2026-03-20 16:26:25.802675 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.802680 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.802684 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.802688 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.802703 | controller | --timeout=300s' 2026-03-20 16:26:25.802711 | controller | - ' ' 2026-03-20 16:26:25.802716 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.802720 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.802724 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.802728 | controller | - ' ' 2026-03-20 16:26:25.802732 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.802737 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.802741 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.802745 | controller | - ' ' 2026-03-20 16:26:25.802749 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.802755 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.802759 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.802764 | controller | available"' 2026-03-20 16:26:25.802768 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.802772 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.802776 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.802781 | controller | - ' exit 1' 2026-03-20 16:26:25.802786 | controller | - ' fi' 2026-03-20 16:26:25.802791 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.802795 | controller | - ' ' 2026-03-20 16:26:25.802799 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.802803 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.802807 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.802811 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.802815 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.802820 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.802824 | controller | -- \' 2026-03-20 16:26:25.802828 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.802832 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.802836 | controller | - ' logger.go:42: 16:15:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.802841 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.802845 | controller | - ' logger.go:42: 16:15:41 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.802849 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.802853 | controller | found' 2026-03-20 16:26:25.802857 | controller | - ' logger.go:42: 16:15:42 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.802861 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.802866 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.802870 | controller | - ' ' 2026-03-20 16:26:25.802874 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.802878 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.802883 | controller | - ' ' 2026-03-20 16:26:25.802887 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.802891 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.802895 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.802900 | controller | - ' exit 1' 2026-03-20 16:26:25.802904 | controller | - ' fi' 2026-03-20 16:26:25.802908 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.802912 | controller | - ' ' 2026-03-20 16:26:25.802916 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.802920 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.802925 | controller | | base64 -d)' 2026-03-20 16:26:25.802929 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.802933 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.802937 | controller | - ' exit 1' 2026-03-20 16:26:25.802944 | controller | - ' fi' 2026-03-20 16:26:25.802949 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.802953 | controller | - ' ' 2026-03-20 16:26:25.802957 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.802969 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.802973 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.802977 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.802982 | controller | --timeout=300s' 2026-03-20 16:26:25.802986 | controller | - ' ' 2026-03-20 16:26:25.802990 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.802994 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.802999 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.803003 | controller | - ' ' 2026-03-20 16:26:25.803007 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.803011 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.803015 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.803020 | controller | - ' ' 2026-03-20 16:26:25.803024 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.803028 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.803032 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.803037 | controller | available"' 2026-03-20 16:26:25.803041 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.803045 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.803049 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.803053 | controller | - ' exit 1' 2026-03-20 16:26:25.803058 | controller | - ' fi' 2026-03-20 16:26:25.803062 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.803066 | controller | - ' ' 2026-03-20 16:26:25.803070 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.803074 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.803078 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.803083 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.803087 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.803091 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.803095 | controller | -- \' 2026-03-20 16:26:25.803099 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.803103 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.803108 | controller | - ' logger.go:42: 16:15:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.803112 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.803116 | controller | - ' logger.go:42: 16:15:42 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.803120 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.803127 | controller | found' 2026-03-20 16:26:25.803133 | controller | - ' logger.go:42: 16:15:43 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.803137 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.803141 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.803146 | controller | - ' ' 2026-03-20 16:26:25.803150 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.803154 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.803158 | controller | - ' ' 2026-03-20 16:26:25.803162 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.803167 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.803171 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.803175 | controller | - ' exit 1' 2026-03-20 16:26:25.803179 | controller | - ' fi' 2026-03-20 16:26:25.803183 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.803188 | controller | - ' ' 2026-03-20 16:26:25.803192 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.803196 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.803200 | controller | | base64 -d)' 2026-03-20 16:26:25.803204 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.803208 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.803213 | controller | - ' exit 1' 2026-03-20 16:26:25.803217 | controller | - ' fi' 2026-03-20 16:26:25.803223 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.803227 | controller | - ' ' 2026-03-20 16:26:25.803232 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.803244 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.803249 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.803253 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.803257 | controller | --timeout=300s' 2026-03-20 16:26:25.803261 | controller | - ' ' 2026-03-20 16:26:25.803266 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.803270 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.803274 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.803278 | controller | - ' ' 2026-03-20 16:26:25.803282 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.803287 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.803291 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.803295 | controller | - ' ' 2026-03-20 16:26:25.803299 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.803304 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.803308 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.803312 | controller | available"' 2026-03-20 16:26:25.803316 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.803320 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.803325 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.803331 | controller | - ' exit 1' 2026-03-20 16:26:25.803335 | controller | - ' fi' 2026-03-20 16:26:25.803339 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.803343 | controller | - ' ' 2026-03-20 16:26:25.803347 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.803352 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.803356 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.803360 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.803364 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.803369 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.803373 | controller | -- \' 2026-03-20 16:26:25.803377 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.803381 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.803385 | controller | - ' logger.go:42: 16:15:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.803389 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.803394 | controller | - ' logger.go:42: 16:15:43 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.803398 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.803402 | controller | found' 2026-03-20 16:26:25.803406 | controller | - ' logger.go:42: 16:15:44 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.803411 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.803415 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.803419 | controller | - ' ' 2026-03-20 16:26:25.803423 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.803428 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.803432 | controller | - ' ' 2026-03-20 16:26:25.803436 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.803440 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.803444 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.803449 | controller | - ' exit 1' 2026-03-20 16:26:25.803453 | controller | - ' fi' 2026-03-20 16:26:25.803457 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.803461 | controller | - ' ' 2026-03-20 16:26:25.803466 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.803470 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.803474 | controller | | base64 -d)' 2026-03-20 16:26:25.803478 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.803482 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.803487 | controller | - ' exit 1' 2026-03-20 16:26:25.803491 | controller | - ' fi' 2026-03-20 16:26:25.803495 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.803499 | controller | - ' ' 2026-03-20 16:26:25.803504 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.803508 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.803519 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.803523 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.803529 | controller | --timeout=300s' 2026-03-20 16:26:25.803534 | controller | - ' ' 2026-03-20 16:26:25.803538 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.803542 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.803546 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.803550 | controller | - ' ' 2026-03-20 16:26:25.803555 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.803559 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.803563 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.803567 | controller | - ' ' 2026-03-20 16:26:25.803572 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.803576 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.803580 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.803584 | controller | available"' 2026-03-20 16:26:25.803588 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.803593 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.803597 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.803601 | controller | - ' exit 1' 2026-03-20 16:26:25.803605 | controller | - ' fi' 2026-03-20 16:26:25.803609 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.803613 | controller | - ' ' 2026-03-20 16:26:25.803618 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.803622 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.803626 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.803630 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.803634 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.803639 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.803645 | controller | -- \' 2026-03-20 16:26:25.803649 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.803653 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.803658 | controller | - ' logger.go:42: 16:15:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.803662 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.803666 | controller | - ' logger.go:42: 16:15:44 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.803670 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.803674 | controller | found' 2026-03-20 16:26:25.803679 | controller | - ' logger.go:42: 16:15:45 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.803683 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.803687 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.803705 | controller | - ' ' 2026-03-20 16:26:25.803727 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.803732 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.803738 | controller | - ' ' 2026-03-20 16:26:25.803742 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.803746 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.803751 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.803756 | controller | - ' exit 1' 2026-03-20 16:26:25.803761 | controller | - ' fi' 2026-03-20 16:26:25.803765 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.803769 | controller | - ' ' 2026-03-20 16:26:25.803773 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.803779 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.803783 | controller | | base64 -d)' 2026-03-20 16:26:25.803788 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.803792 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.803796 | controller | - ' exit 1' 2026-03-20 16:26:25.803800 | controller | - ' fi' 2026-03-20 16:26:25.803804 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.803808 | controller | - ' ' 2026-03-20 16:26:25.803813 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.803817 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.803828 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.803833 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.803837 | controller | --timeout=300s' 2026-03-20 16:26:25.803841 | controller | - ' ' 2026-03-20 16:26:25.803846 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.803850 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.803854 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.803858 | controller | - ' ' 2026-03-20 16:26:25.803862 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.803866 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.803871 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.803875 | controller | - ' ' 2026-03-20 16:26:25.803879 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.803883 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.803888 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.803892 | controller | available"' 2026-03-20 16:26:25.803896 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.803900 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.803904 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.803908 | controller | - ' exit 1' 2026-03-20 16:26:25.803913 | controller | - ' fi' 2026-03-20 16:26:25.803917 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.803921 | controller | - ' ' 2026-03-20 16:26:25.803925 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.803929 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.803933 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.803939 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.803943 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.803948 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.803952 | controller | -- \' 2026-03-20 16:26:25.803956 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.803960 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.803964 | controller | - ' logger.go:42: 16:15:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.803968 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.803973 | controller | - ' logger.go:42: 16:15:45 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.803977 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.803981 | controller | found' 2026-03-20 16:26:25.803985 | controller | - ' logger.go:42: 16:15:46 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.803989 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.803993 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.803998 | controller | - ' ' 2026-03-20 16:26:25.804002 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.804006 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.804010 | controller | - ' ' 2026-03-20 16:26:25.804014 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.804019 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.804023 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.804027 | controller | - ' exit 1' 2026-03-20 16:26:25.804031 | controller | - ' fi' 2026-03-20 16:26:25.804036 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.804040 | controller | - ' ' 2026-03-20 16:26:25.804044 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.804048 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.804052 | controller | | base64 -d)' 2026-03-20 16:26:25.804056 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.804061 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.804065 | controller | - ' exit 1' 2026-03-20 16:26:25.804069 | controller | - ' fi' 2026-03-20 16:26:25.804073 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.804077 | controller | - ' ' 2026-03-20 16:26:25.804081 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.804086 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.804090 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.804103 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.804109 | controller | --timeout=300s' 2026-03-20 16:26:25.804114 | controller | - ' ' 2026-03-20 16:26:25.804119 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.804123 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.804127 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.804131 | controller | - ' ' 2026-03-20 16:26:25.804138 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.804142 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.804146 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.804150 | controller | - ' ' 2026-03-20 16:26:25.804154 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.804158 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.804163 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.804167 | controller | available"' 2026-03-20 16:26:25.804171 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.804175 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.804180 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.804184 | controller | - ' exit 1' 2026-03-20 16:26:25.804188 | controller | - ' fi' 2026-03-20 16:26:25.804192 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.804196 | controller | - ' ' 2026-03-20 16:26:25.804200 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.804205 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.804209 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.804213 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.804217 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.804221 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.804227 | controller | -- \' 2026-03-20 16:26:25.804232 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.804237 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.804241 | controller | - ' logger.go:42: 16:15:46 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.804245 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.804250 | controller | - ' logger.go:42: 16:15:46 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.804254 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.804258 | controller | found' 2026-03-20 16:26:25.804263 | controller | - ' logger.go:42: 16:15:48 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.804269 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.804274 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.804280 | controller | - ' ' 2026-03-20 16:26:25.804286 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.804295 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.804302 | controller | - ' ' 2026-03-20 16:26:25.804309 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.804316 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.804323 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.804331 | controller | - ' exit 1' 2026-03-20 16:26:25.804338 | controller | - ' fi' 2026-03-20 16:26:25.804343 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.804347 | controller | - ' ' 2026-03-20 16:26:25.804351 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.804359 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.804363 | controller | | base64 -d)' 2026-03-20 16:26:25.804368 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.804372 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.804376 | controller | - ' exit 1' 2026-03-20 16:26:25.804380 | controller | - ' fi' 2026-03-20 16:26:25.804385 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.804389 | controller | - ' ' 2026-03-20 16:26:25.804393 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.804397 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.804401 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.804406 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.804421 | controller | --timeout=300s' 2026-03-20 16:26:25.804428 | controller | - ' ' 2026-03-20 16:26:25.804433 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.804439 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.804445 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.804450 | controller | - ' ' 2026-03-20 16:26:25.804455 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.804461 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.804466 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.804472 | controller | - ' ' 2026-03-20 16:26:25.804477 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.804483 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.804488 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.804493 | controller | available"' 2026-03-20 16:26:25.804499 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.804505 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.804510 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.804515 | controller | - ' exit 1' 2026-03-20 16:26:25.804520 | controller | - ' fi' 2026-03-20 16:26:25.804526 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.804531 | controller | - ' ' 2026-03-20 16:26:25.804537 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.804542 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.804548 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.804553 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.804559 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.804565 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.804571 | controller | -- \' 2026-03-20 16:26:25.804577 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.804585 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.804589 | controller | - ' logger.go:42: 16:15:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.804594 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.804598 | controller | - ' logger.go:42: 16:15:48 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.804602 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.804606 | controller | found' 2026-03-20 16:26:25.804610 | controller | - ' logger.go:42: 16:15:49 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.804615 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.804619 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.804623 | controller | - ' ' 2026-03-20 16:26:25.804627 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.804631 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.804636 | controller | - ' ' 2026-03-20 16:26:25.804640 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.804644 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.804648 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.804653 | controller | - ' exit 1' 2026-03-20 16:26:25.804657 | controller | - ' fi' 2026-03-20 16:26:25.804661 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.804665 | controller | - ' ' 2026-03-20 16:26:25.804669 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.804674 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.804678 | controller | | base64 -d)' 2026-03-20 16:26:25.804682 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.804686 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.804690 | controller | - ' exit 1' 2026-03-20 16:26:25.804712 | controller | - ' fi' 2026-03-20 16:26:25.804717 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.804721 | controller | - ' ' 2026-03-20 16:26:25.804725 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.804730 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.804734 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.804738 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.804742 | controller | --timeout=300s' 2026-03-20 16:26:25.804746 | controller | - ' ' 2026-03-20 16:26:25.804760 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.804765 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.804769 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.804773 | controller | - ' ' 2026-03-20 16:26:25.804777 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.804782 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.804789 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.804793 | controller | - ' ' 2026-03-20 16:26:25.804797 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.804802 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.804808 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.804812 | controller | available"' 2026-03-20 16:26:25.804816 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.804820 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.804824 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.804830 | controller | - ' exit 1' 2026-03-20 16:26:25.804834 | controller | - ' fi' 2026-03-20 16:26:25.804839 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.804843 | controller | - ' ' 2026-03-20 16:26:25.804847 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.804851 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.804855 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.804860 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.804864 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.804868 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.804872 | controller | -- \' 2026-03-20 16:26:25.804876 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.804881 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.804885 | controller | - ' logger.go:42: 16:15:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.804889 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.804893 | controller | - ' logger.go:42: 16:15:49 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.804897 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.804902 | controller | found' 2026-03-20 16:26:25.804906 | controller | - ' logger.go:42: 16:15:50 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.804910 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.804914 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.804919 | controller | - ' ' 2026-03-20 16:26:25.804923 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.804927 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.804931 | controller | - ' ' 2026-03-20 16:26:25.804937 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.804941 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.804946 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.804950 | controller | - ' exit 1' 2026-03-20 16:26:25.804954 | controller | - ' fi' 2026-03-20 16:26:25.804958 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.804963 | controller | - ' ' 2026-03-20 16:26:25.804967 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.804971 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.804975 | controller | | base64 -d)' 2026-03-20 16:26:25.804979 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.804984 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.804988 | controller | - ' exit 1' 2026-03-20 16:26:25.804992 | controller | - ' fi' 2026-03-20 16:26:25.804996 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.805002 | controller | - ' ' 2026-03-20 16:26:25.805007 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.805012 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.805016 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.805020 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.805024 | controller | --timeout=300s' 2026-03-20 16:26:25.805028 | controller | - ' ' 2026-03-20 16:26:25.805040 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.805044 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.805048 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.805053 | controller | - ' ' 2026-03-20 16:26:25.805057 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.805061 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.805066 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.805070 | controller | - ' ' 2026-03-20 16:26:25.805074 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.805078 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.805085 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.805090 | controller | available"' 2026-03-20 16:26:25.805094 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.805098 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.805104 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.805108 | controller | - ' exit 1' 2026-03-20 16:26:25.805113 | controller | - ' fi' 2026-03-20 16:26:25.805117 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.805121 | controller | - ' ' 2026-03-20 16:26:25.805125 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.805129 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.805134 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.805138 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.805142 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.805146 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.805150 | controller | -- \' 2026-03-20 16:26:25.805155 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.805159 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.805163 | controller | - ' logger.go:42: 16:15:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.805167 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.805172 | controller | - ' logger.go:42: 16:15:50 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.805176 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.805180 | controller | found' 2026-03-20 16:26:25.805186 | controller | - ' logger.go:42: 16:15:51 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.805190 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.805194 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.805199 | controller | - ' ' 2026-03-20 16:26:25.805203 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.805207 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.805211 | controller | - ' ' 2026-03-20 16:26:25.805215 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.805219 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.805224 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.805228 | controller | - ' exit 1' 2026-03-20 16:26:25.805232 | controller | - ' fi' 2026-03-20 16:26:25.805236 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.805241 | controller | - ' ' 2026-03-20 16:26:25.805247 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.805258 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.805271 | controller | | base64 -d)' 2026-03-20 16:26:25.805277 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.805282 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.805288 | controller | - ' exit 1' 2026-03-20 16:26:25.805294 | controller | - ' fi' 2026-03-20 16:26:25.805300 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.805305 | controller | - ' ' 2026-03-20 16:26:25.805311 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.805317 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.805323 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.805328 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.805334 | controller | --timeout=300s' 2026-03-20 16:26:25.805340 | controller | - ' ' 2026-03-20 16:26:25.805345 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.805365 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.805377 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.805390 | controller | - ' ' 2026-03-20 16:26:25.805397 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.805403 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.805409 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.805415 | controller | - ' ' 2026-03-20 16:26:25.805420 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.805426 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.805432 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.805438 | controller | available"' 2026-03-20 16:26:25.805444 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.805450 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.805456 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.805462 | controller | - ' exit 1' 2026-03-20 16:26:25.805474 | controller | - ' fi' 2026-03-20 16:26:25.805480 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.805485 | controller | - ' ' 2026-03-20 16:26:25.805491 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.805496 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.805502 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.805508 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.805514 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.805520 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.805525 | controller | -- \' 2026-03-20 16:26:25.805531 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.805537 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.805543 | controller | - ' logger.go:42: 16:15:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.805549 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.805555 | controller | - ' logger.go:42: 16:15:51 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.805561 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.805566 | controller | found' 2026-03-20 16:26:25.805572 | controller | - ' logger.go:42: 16:15:52 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.805577 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.805583 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.805588 | controller | - ' ' 2026-03-20 16:26:25.805593 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.805599 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.805604 | controller | - ' ' 2026-03-20 16:26:25.805610 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.805614 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.805618 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.805622 | controller | - ' exit 1' 2026-03-20 16:26:25.805626 | controller | - ' fi' 2026-03-20 16:26:25.805631 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.805638 | controller | - ' ' 2026-03-20 16:26:25.805642 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.805646 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.805651 | controller | | base64 -d)' 2026-03-20 16:26:25.805655 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.805659 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.805663 | controller | - ' exit 1' 2026-03-20 16:26:25.805668 | controller | - ' fi' 2026-03-20 16:26:25.805672 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.805676 | controller | - ' ' 2026-03-20 16:26:25.805680 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.805685 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.805689 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.805709 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.805716 | controller | --timeout=300s' 2026-03-20 16:26:25.805724 | controller | - ' ' 2026-03-20 16:26:25.805729 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.805743 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.805747 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.805752 | controller | - ' ' 2026-03-20 16:26:25.805756 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.805761 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.805765 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.805770 | controller | - ' ' 2026-03-20 16:26:25.805775 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.805784 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.805798 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.805805 | controller | available"' 2026-03-20 16:26:25.805811 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.805817 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.805823 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.805829 | controller | - ' exit 1' 2026-03-20 16:26:25.805835 | controller | - ' fi' 2026-03-20 16:26:25.805841 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.805848 | controller | - ' ' 2026-03-20 16:26:25.805854 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.805859 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.805866 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.805872 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.805878 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.805883 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.805889 | controller | -- \' 2026-03-20 16:26:25.805895 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.805901 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.805911 | controller | - ' logger.go:42: 16:15:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.805917 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.805923 | controller | - ' logger.go:42: 16:15:53 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.805929 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.805935 | controller | found' 2026-03-20 16:26:25.805941 | controller | - ' logger.go:42: 16:15:54 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.805946 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.805952 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.805958 | controller | - ' ' 2026-03-20 16:26:25.805964 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.805969 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.805975 | controller | - ' ' 2026-03-20 16:26:25.805986 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.805992 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.805998 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.806004 | controller | - ' exit 1' 2026-03-20 16:26:25.806009 | controller | - ' fi' 2026-03-20 16:26:25.806015 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.806021 | controller | - ' ' 2026-03-20 16:26:25.806027 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.806032 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.806039 | controller | | base64 -d)' 2026-03-20 16:26:25.806044 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.806050 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.806056 | controller | - ' exit 1' 2026-03-20 16:26:25.806061 | controller | - ' fi' 2026-03-20 16:26:25.806067 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.806073 | controller | - ' ' 2026-03-20 16:26:25.806079 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.806084 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.806090 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.806096 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.806102 | controller | --timeout=300s' 2026-03-20 16:26:25.806107 | controller | - ' ' 2026-03-20 16:26:25.806113 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.806119 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.806137 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.806145 | controller | - ' ' 2026-03-20 16:26:25.806151 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.806157 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.806162 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.806168 | controller | - ' ' 2026-03-20 16:26:25.806174 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.806180 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.806186 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.806192 | controller | available"' 2026-03-20 16:26:25.806198 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.806204 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.806209 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.806215 | controller | - ' exit 1' 2026-03-20 16:26:25.806221 | controller | - ' fi' 2026-03-20 16:26:25.806227 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.806233 | controller | - ' ' 2026-03-20 16:26:25.806238 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.806244 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.806250 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.806256 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.806265 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.806271 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.806277 | controller | -- \' 2026-03-20 16:26:25.806283 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.806289 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.806294 | controller | - ' logger.go:42: 16:15:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.806300 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.806306 | controller | - ' logger.go:42: 16:15:54 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.806312 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.806317 | controller | found' 2026-03-20 16:26:25.806323 | controller | - ' logger.go:42: 16:15:55 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.806329 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.806335 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.806344 | controller | - ' ' 2026-03-20 16:26:25.806350 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.806355 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.806361 | controller | - ' ' 2026-03-20 16:26:25.806367 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.806373 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.806379 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.806385 | controller | - ' exit 1' 2026-03-20 16:26:25.806391 | controller | - ' fi' 2026-03-20 16:26:25.806396 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.806402 | controller | - ' ' 2026-03-20 16:26:25.806408 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.806414 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.806420 | controller | | base64 -d)' 2026-03-20 16:26:25.806426 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.806431 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.806437 | controller | - ' exit 1' 2026-03-20 16:26:25.806443 | controller | - ' fi' 2026-03-20 16:26:25.806449 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.806454 | controller | - ' ' 2026-03-20 16:26:25.806460 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.806466 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.806475 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.806481 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.806486 | controller | --timeout=300s' 2026-03-20 16:26:25.806492 | controller | - ' ' 2026-03-20 16:26:25.806498 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.806504 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.806520 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.806528 | controller | - ' ' 2026-03-20 16:26:25.806534 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.806543 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.806549 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.806555 | controller | - ' ' 2026-03-20 16:26:25.806561 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.806567 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.806573 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.806579 | controller | available"' 2026-03-20 16:26:25.806585 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.806591 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.806598 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.806604 | controller | - ' exit 1' 2026-03-20 16:26:25.806610 | controller | - ' fi' 2026-03-20 16:26:25.806617 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.806623 | controller | - ' ' 2026-03-20 16:26:25.806629 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.806634 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.806640 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.806647 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.806653 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.806659 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.806665 | controller | -- \' 2026-03-20 16:26:25.806671 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.806677 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.806683 | controller | - ' logger.go:42: 16:15:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.806688 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.806721 | controller | - ' logger.go:42: 16:15:55 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.806727 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.806733 | controller | found' 2026-03-20 16:26:25.806738 | controller | - ' logger.go:42: 16:15:56 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.806744 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.806750 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.806756 | controller | - ' ' 2026-03-20 16:26:25.806761 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.806767 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.806773 | controller | - ' ' 2026-03-20 16:26:25.806779 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.806784 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.806790 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.806796 | controller | - ' exit 1' 2026-03-20 16:26:25.806801 | controller | - ' fi' 2026-03-20 16:26:25.806807 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.806813 | controller | - ' ' 2026-03-20 16:26:25.806823 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.806844 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.806850 | controller | | base64 -d)' 2026-03-20 16:26:25.806856 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.806862 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.806868 | controller | - ' exit 1' 2026-03-20 16:26:25.806873 | controller | - ' fi' 2026-03-20 16:26:25.806879 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.806887 | controller | - ' ' 2026-03-20 16:26:25.806893 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.806899 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.806905 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.806910 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.806916 | controller | --timeout=300s' 2026-03-20 16:26:25.806922 | controller | - ' ' 2026-03-20 16:26:25.806930 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.806936 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.806953 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.806959 | controller | - ' ' 2026-03-20 16:26:25.806965 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.806970 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.806976 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.806982 | controller | - ' ' 2026-03-20 16:26:25.806987 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.806993 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.806999 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.807005 | controller | available"' 2026-03-20 16:26:25.807011 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.807017 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.807030 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.807040 | controller | - ' exit 1' 2026-03-20 16:26:25.807047 | controller | - ' fi' 2026-03-20 16:26:25.807053 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.807059 | controller | - ' ' 2026-03-20 16:26:25.807065 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.807070 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.807075 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.807079 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.807083 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.807087 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.807092 | controller | -- \' 2026-03-20 16:26:25.807096 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.807100 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.807108 | controller | - ' logger.go:42: 16:15:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.807112 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.807116 | controller | - ' logger.go:42: 16:15:56 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.807120 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.807125 | controller | found' 2026-03-20 16:26:25.807129 | controller | - ' logger.go:42: 16:15:57 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.807133 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.807137 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.807141 | controller | - ' ' 2026-03-20 16:26:25.807145 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.807150 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.807154 | controller | - ' ' 2026-03-20 16:26:25.807158 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.807162 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.807166 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.807170 | controller | - ' exit 1' 2026-03-20 16:26:25.807175 | controller | - ' fi' 2026-03-20 16:26:25.807179 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.807183 | controller | - ' ' 2026-03-20 16:26:25.807187 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.807191 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.807196 | controller | | base64 -d)' 2026-03-20 16:26:25.807200 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.807204 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.807208 | controller | - ' exit 1' 2026-03-20 16:26:25.807212 | controller | - ' fi' 2026-03-20 16:26:25.807216 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.807220 | controller | - ' ' 2026-03-20 16:26:25.807225 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.807229 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.807233 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.807250 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.807255 | controller | --timeout=300s' 2026-03-20 16:26:25.807261 | controller | - ' ' 2026-03-20 16:26:25.807265 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.807269 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.807273 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.807278 | controller | - ' ' 2026-03-20 16:26:25.807292 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.807297 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.807301 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.807305 | controller | - ' ' 2026-03-20 16:26:25.807310 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.807314 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.807320 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.807324 | controller | available"' 2026-03-20 16:26:25.807329 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.807333 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.807337 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.807341 | controller | - ' exit 1' 2026-03-20 16:26:25.807346 | controller | - ' fi' 2026-03-20 16:26:25.807350 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.807354 | controller | - ' ' 2026-03-20 16:26:25.807360 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.807370 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.807383 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.807390 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.807394 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.807398 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.807402 | controller | -- \' 2026-03-20 16:26:25.807407 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.807411 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.807415 | controller | - ' logger.go:42: 16:15:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.807421 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.807431 | controller | - ' logger.go:42: 16:15:57 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.807437 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.807443 | controller | found' 2026-03-20 16:26:25.807449 | controller | - ' logger.go:42: 16:15:58 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.807455 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.807461 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.807466 | controller | - ' ' 2026-03-20 16:26:25.807472 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.807478 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.807484 | controller | - ' ' 2026-03-20 16:26:25.807490 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.807496 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.807502 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.807508 | controller | - ' exit 1' 2026-03-20 16:26:25.807514 | controller | - ' fi' 2026-03-20 16:26:25.807520 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.807525 | controller | - ' ' 2026-03-20 16:26:25.807531 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.807537 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.807543 | controller | | base64 -d)' 2026-03-20 16:26:25.807549 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.807555 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.807560 | controller | - ' exit 1' 2026-03-20 16:26:25.807566 | controller | - ' fi' 2026-03-20 16:26:25.807571 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.807580 | controller | - ' ' 2026-03-20 16:26:25.807586 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.807591 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.807597 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.807609 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.807621 | controller | --timeout=300s' 2026-03-20 16:26:25.807627 | controller | - ' ' 2026-03-20 16:26:25.807633 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.807639 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.807645 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.807650 | controller | - ' ' 2026-03-20 16:26:25.807665 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.807671 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.807675 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.807679 | controller | - ' ' 2026-03-20 16:26:25.807684 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.807688 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.807712 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.807737 | controller | available"' 2026-03-20 16:26:25.807742 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.807746 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.807750 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.807755 | controller | - ' exit 1' 2026-03-20 16:26:25.807760 | controller | - ' fi' 2026-03-20 16:26:25.807773 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.807783 | controller | - ' ' 2026-03-20 16:26:25.807790 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.807796 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.807802 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.807808 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.807813 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.807819 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.807825 | controller | -- \' 2026-03-20 16:26:25.807832 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.807838 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.807844 | controller | - ' logger.go:42: 16:15:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.807850 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.807856 | controller | - ' logger.go:42: 16:15:58 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.807862 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.807868 | controller | found' 2026-03-20 16:26:25.807875 | controller | - ' logger.go:42: 16:15:59 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.807888 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.807894 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.807900 | controller | - ' ' 2026-03-20 16:26:25.807906 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.807912 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.807918 | controller | - ' ' 2026-03-20 16:26:25.807924 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.807930 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.807936 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.807941 | controller | - ' exit 1' 2026-03-20 16:26:25.807947 | controller | - ' fi' 2026-03-20 16:26:25.807953 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.807959 | controller | - ' ' 2026-03-20 16:26:25.807965 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.807970 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.807975 | controller | | base64 -d)' 2026-03-20 16:26:25.807979 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.807983 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.807988 | controller | - ' exit 1' 2026-03-20 16:26:25.807992 | controller | - ' fi' 2026-03-20 16:26:25.807996 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.808000 | controller | - ' ' 2026-03-20 16:26:25.808004 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.808009 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.808013 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.808017 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.808021 | controller | --timeout=300s' 2026-03-20 16:26:25.808025 | controller | - ' ' 2026-03-20 16:26:25.808029 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.808034 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.808038 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.808042 | controller | - ' ' 2026-03-20 16:26:25.808046 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.808067 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.808072 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.808078 | controller | - ' ' 2026-03-20 16:26:25.808094 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.808103 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.808109 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.808115 | controller | available"' 2026-03-20 16:26:25.808121 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.808127 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.808133 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.808138 | controller | - ' exit 1' 2026-03-20 16:26:25.808142 | controller | - ' fi' 2026-03-20 16:26:25.808146 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.808153 | controller | - ' ' 2026-03-20 16:26:25.808158 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.808162 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.808166 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.808170 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.808175 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.808179 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.808183 | controller | -- \' 2026-03-20 16:26:25.808187 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.808192 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.808196 | controller | - ' logger.go:42: 16:15:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.808200 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.808204 | controller | - ' logger.go:42: 16:15:59 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.808209 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.808213 | controller | found' 2026-03-20 16:26:25.808217 | controller | - ' logger.go:42: 16:16:00 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.808221 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.808225 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.808230 | controller | - ' ' 2026-03-20 16:26:25.808234 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.808238 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.808242 | controller | - ' ' 2026-03-20 16:26:25.808246 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.808250 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.808255 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.808259 | controller | - ' exit 1' 2026-03-20 16:26:25.808263 | controller | - ' fi' 2026-03-20 16:26:25.808267 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.808272 | controller | - ' ' 2026-03-20 16:26:25.808279 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.808283 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.808287 | controller | | base64 -d)' 2026-03-20 16:26:25.808292 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.808296 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.808300 | controller | - ' exit 1' 2026-03-20 16:26:25.808304 | controller | - ' fi' 2026-03-20 16:26:25.808308 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.808313 | controller | - ' ' 2026-03-20 16:26:25.808317 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.808321 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.808325 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.808329 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.808334 | controller | --timeout=300s' 2026-03-20 16:26:25.808338 | controller | - ' ' 2026-03-20 16:26:25.808342 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.808348 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.808352 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.808356 | controller | - ' ' 2026-03-20 16:26:25.808361 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.808377 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.808382 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.808387 | controller | - ' ' 2026-03-20 16:26:25.808392 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.808398 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.808413 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.808422 | controller | available"' 2026-03-20 16:26:25.808427 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.808435 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.808439 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.808443 | controller | - ' exit 1' 2026-03-20 16:26:25.808448 | controller | - ' fi' 2026-03-20 16:26:25.808452 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.808456 | controller | - ' ' 2026-03-20 16:26:25.808460 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.808464 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.808469 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.808473 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.808477 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.808481 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.808487 | controller | -- \' 2026-03-20 16:26:25.808493 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.808498 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.808504 | controller | - ' logger.go:42: 16:16:00 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.808510 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.808515 | controller | - ' logger.go:42: 16:16:01 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.808521 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.808525 | controller | found' 2026-03-20 16:26:25.808529 | controller | - ' logger.go:42: 16:16:02 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.808533 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.808537 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.808542 | controller | - ' ' 2026-03-20 16:26:25.808546 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.808550 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.808554 | controller | - ' ' 2026-03-20 16:26:25.808558 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.808565 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.808570 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.808576 | controller | - ' exit 1' 2026-03-20 16:26:25.808580 | controller | - ' fi' 2026-03-20 16:26:25.808584 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.808588 | controller | - ' ' 2026-03-20 16:26:25.808593 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.808597 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.808601 | controller | | base64 -d)' 2026-03-20 16:26:25.808605 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.808609 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.808614 | controller | - ' exit 1' 2026-03-20 16:26:25.808618 | controller | - ' fi' 2026-03-20 16:26:25.808622 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.808626 | controller | - ' ' 2026-03-20 16:26:25.808630 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.808634 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.808639 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.808643 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.808647 | controller | --timeout=300s' 2026-03-20 16:26:25.808651 | controller | - ' ' 2026-03-20 16:26:25.808655 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.808660 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.808665 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.808670 | controller | - ' ' 2026-03-20 16:26:25.808674 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.808679 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.808709 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.808718 | controller | - ' ' 2026-03-20 16:26:25.808724 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.808728 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.808735 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.808739 | controller | available"' 2026-03-20 16:26:25.808744 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.808748 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.808752 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.808759 | controller | - ' exit 1' 2026-03-20 16:26:25.808763 | controller | - ' fi' 2026-03-20 16:26:25.808767 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.808771 | controller | - ' ' 2026-03-20 16:26:25.808776 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.808780 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.808784 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.808788 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.808794 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.808798 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.808803 | controller | -- \' 2026-03-20 16:26:25.808808 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.808814 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.808820 | controller | - ' logger.go:42: 16:16:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.808826 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.808831 | controller | - ' logger.go:42: 16:16:02 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.808836 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.808840 | controller | found' 2026-03-20 16:26:25.808844 | controller | - ' logger.go:42: 16:16:03 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.808848 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.808853 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.808857 | controller | - ' ' 2026-03-20 16:26:25.808861 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.808866 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.808871 | controller | - ' ' 2026-03-20 16:26:25.808877 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.808883 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.808889 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.808895 | controller | - ' exit 1' 2026-03-20 16:26:25.808899 | controller | - ' fi' 2026-03-20 16:26:25.808904 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.808909 | controller | - ' ' 2026-03-20 16:26:25.808914 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.808920 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.808925 | controller | | base64 -d)' 2026-03-20 16:26:25.808931 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.808937 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.808943 | controller | - ' exit 1' 2026-03-20 16:26:25.808949 | controller | - ' fi' 2026-03-20 16:26:25.808954 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.808959 | controller | - ' ' 2026-03-20 16:26:25.808963 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.808968 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.808972 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.808976 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.808980 | controller | --timeout=300s' 2026-03-20 16:26:25.808984 | controller | - ' ' 2026-03-20 16:26:25.808989 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.808993 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.808997 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.809001 | controller | - ' ' 2026-03-20 16:26:25.809006 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.809012 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.809026 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.809030 | controller | - ' ' 2026-03-20 16:26:25.809035 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.809040 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.809044 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.809049 | controller | available"' 2026-03-20 16:26:25.809053 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.809057 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.809061 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.809065 | controller | - ' exit 1' 2026-03-20 16:26:25.809070 | controller | - ' fi' 2026-03-20 16:26:25.809074 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.809078 | controller | - ' ' 2026-03-20 16:26:25.809082 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.809086 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.809091 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.809095 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.809099 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.809103 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.809107 | controller | -- \' 2026-03-20 16:26:25.809112 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.809116 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.809120 | controller | - ' logger.go:42: 16:16:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.809124 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.809128 | controller | - ' logger.go:42: 16:16:03 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.809132 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.809137 | controller | found' 2026-03-20 16:26:25.809141 | controller | - ' logger.go:42: 16:16:04 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.809145 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.809149 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.809153 | controller | - ' ' 2026-03-20 16:26:25.809158 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.809162 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.809166 | controller | - ' ' 2026-03-20 16:26:25.809170 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.809174 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.809178 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.809183 | controller | - ' exit 1' 2026-03-20 16:26:25.809187 | controller | - ' fi' 2026-03-20 16:26:25.809191 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.809195 | controller | - ' ' 2026-03-20 16:26:25.809199 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.809204 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.809210 | controller | | base64 -d)' 2026-03-20 16:26:25.809214 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.809218 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.809222 | controller | - ' exit 1' 2026-03-20 16:26:25.809227 | controller | - ' fi' 2026-03-20 16:26:25.809231 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.809235 | controller | - ' ' 2026-03-20 16:26:25.809239 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.809243 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.809247 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.809252 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.809256 | controller | --timeout=300s' 2026-03-20 16:26:25.809260 | controller | - ' ' 2026-03-20 16:26:25.809264 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.809268 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.809273 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.809279 | controller | - ' ' 2026-03-20 16:26:25.809285 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.809290 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.809296 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.809309 | controller | - ' ' 2026-03-20 16:26:25.809314 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.809319 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.809323 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.809327 | controller | available"' 2026-03-20 16:26:25.809331 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.809339 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.809345 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.809351 | controller | - ' exit 1' 2026-03-20 16:26:25.809356 | controller | - ' fi' 2026-03-20 16:26:25.809362 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.809368 | controller | - ' ' 2026-03-20 16:26:25.809377 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.809382 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.809386 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.809390 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.809394 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.809398 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.809402 | controller | -- \' 2026-03-20 16:26:25.809407 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.809411 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.809417 | controller | - ' logger.go:42: 16:16:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.809421 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.809425 | controller | - ' logger.go:42: 16:16:04 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.809429 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.809434 | controller | found' 2026-03-20 16:26:25.809440 | controller | - ' logger.go:42: 16:16:05 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.809446 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.809451 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.809457 | controller | - ' ' 2026-03-20 16:26:25.809463 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.809469 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.809474 | controller | - ' ' 2026-03-20 16:26:25.809480 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.809484 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.809488 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.809493 | controller | - ' exit 1' 2026-03-20 16:26:25.809497 | controller | - ' fi' 2026-03-20 16:26:25.809501 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.809505 | controller | - ' ' 2026-03-20 16:26:25.809509 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.809514 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.809518 | controller | | base64 -d)' 2026-03-20 16:26:25.809522 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.809526 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.809530 | controller | - ' exit 1' 2026-03-20 16:26:25.809534 | controller | - ' fi' 2026-03-20 16:26:25.809539 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.809543 | controller | - ' ' 2026-03-20 16:26:25.809547 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.809551 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.809555 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.809559 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.809563 | controller | --timeout=300s' 2026-03-20 16:26:25.809568 | controller | - ' ' 2026-03-20 16:26:25.809572 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.809576 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.809580 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.809584 | controller | - ' ' 2026-03-20 16:26:25.809588 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.809592 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.809596 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.809601 | controller | - ' ' 2026-03-20 16:26:25.809614 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.809618 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.809623 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.809630 | controller | available"' 2026-03-20 16:26:25.809635 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.809639 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.809643 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.809647 | controller | - ' exit 1' 2026-03-20 16:26:25.809651 | controller | - ' fi' 2026-03-20 16:26:25.809656 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.809660 | controller | - ' ' 2026-03-20 16:26:25.809664 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.809668 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.809672 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.809676 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.809683 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.809687 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.809708 | controller | -- \' 2026-03-20 16:26:25.809715 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.809719 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.809724 | controller | - ' logger.go:42: 16:16:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.809728 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.809732 | controller | - ' logger.go:42: 16:16:05 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.809736 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.809740 | controller | found' 2026-03-20 16:26:25.809744 | controller | - ' logger.go:42: 16:16:06 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.809748 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.809753 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.809757 | controller | - ' ' 2026-03-20 16:26:25.809761 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.809765 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.809769 | controller | - ' ' 2026-03-20 16:26:25.809773 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.809778 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.809782 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.809786 | controller | - ' exit 1' 2026-03-20 16:26:25.809790 | controller | - ' fi' 2026-03-20 16:26:25.809794 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.809798 | controller | - ' ' 2026-03-20 16:26:25.809805 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.809809 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.809813 | controller | | base64 -d)' 2026-03-20 16:26:25.809817 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.809821 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.809826 | controller | - ' exit 1' 2026-03-20 16:26:25.809830 | controller | - ' fi' 2026-03-20 16:26:25.809834 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.809838 | controller | - ' ' 2026-03-20 16:26:25.809842 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.809850 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.809856 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.809861 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.809867 | controller | --timeout=300s' 2026-03-20 16:26:25.809873 | controller | - ' ' 2026-03-20 16:26:25.809877 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.809881 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.809885 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.809889 | controller | - ' ' 2026-03-20 16:26:25.809894 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.809898 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.809902 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.809908 | controller | - ' ' 2026-03-20 16:26:25.809914 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.809930 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.809939 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.809943 | controller | available"' 2026-03-20 16:26:25.809947 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.809951 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.809956 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.809960 | controller | - ' exit 1' 2026-03-20 16:26:25.809964 | controller | - ' fi' 2026-03-20 16:26:25.809968 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.809972 | controller | - ' ' 2026-03-20 16:26:25.809977 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.809981 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.809987 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.809991 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.809995 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.809999 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.810003 | controller | -- \' 2026-03-20 16:26:25.810008 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.810012 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.810016 | controller | - ' logger.go:42: 16:16:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.810020 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.810024 | controller | - ' logger.go:42: 16:16:06 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.810029 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.810033 | controller | found' 2026-03-20 16:26:25.810037 | controller | - ' logger.go:42: 16:16:07 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.810044 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.810048 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.810053 | controller | - ' ' 2026-03-20 16:26:25.810057 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.810061 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.810065 | controller | - ' ' 2026-03-20 16:26:25.810070 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.810074 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.810078 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.810082 | controller | - ' exit 1' 2026-03-20 16:26:25.810086 | controller | - ' fi' 2026-03-20 16:26:25.810091 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.810095 | controller | - ' ' 2026-03-20 16:26:25.810099 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.810103 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.810107 | controller | | base64 -d)' 2026-03-20 16:26:25.810112 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.810116 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.810120 | controller | - ' exit 1' 2026-03-20 16:26:25.810124 | controller | - ' fi' 2026-03-20 16:26:25.810128 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.810133 | controller | - ' ' 2026-03-20 16:26:25.810137 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.810141 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.810145 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.810149 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.810153 | controller | --timeout=300s' 2026-03-20 16:26:25.810158 | controller | - ' ' 2026-03-20 16:26:25.810162 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.810166 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.810170 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.810174 | controller | - ' ' 2026-03-20 16:26:25.810179 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.810184 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.810190 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.810196 | controller | - ' ' 2026-03-20 16:26:25.810202 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.810215 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.810220 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.810224 | controller | available"' 2026-03-20 16:26:25.810228 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.810232 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.810237 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.810241 | controller | - ' exit 1' 2026-03-20 16:26:25.810245 | controller | - ' fi' 2026-03-20 16:26:25.810249 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.810255 | controller | - ' ' 2026-03-20 16:26:25.810260 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.810264 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.810268 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.810273 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.810278 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.810284 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.810290 | controller | -- \' 2026-03-20 16:26:25.810296 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.810301 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.810307 | controller | - ' logger.go:42: 16:16:07 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.810311 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.810315 | controller | - ' logger.go:42: 16:16:07 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.810320 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.810324 | controller | found' 2026-03-20 16:26:25.810328 | controller | - ' logger.go:42: 16:16:08 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.810332 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.810336 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.810341 | controller | - ' ' 2026-03-20 16:26:25.810345 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.810349 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.810353 | controller | - ' ' 2026-03-20 16:26:25.810357 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.810361 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.810366 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.810370 | controller | - ' exit 1' 2026-03-20 16:26:25.810374 | controller | - ' fi' 2026-03-20 16:26:25.810378 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.810383 | controller | - ' ' 2026-03-20 16:26:25.810389 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.810394 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.810398 | controller | | base64 -d)' 2026-03-20 16:26:25.810402 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.810406 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.810410 | controller | - ' exit 1' 2026-03-20 16:26:25.810414 | controller | - ' fi' 2026-03-20 16:26:25.810419 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.810423 | controller | - ' ' 2026-03-20 16:26:25.810427 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.810431 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.810435 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.810439 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.810445 | controller | --timeout=300s' 2026-03-20 16:26:25.810451 | controller | - ' ' 2026-03-20 16:26:25.810456 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.810465 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.810490 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.810495 | controller | - ' ' 2026-03-20 16:26:25.810501 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.810505 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.810509 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.810514 | controller | - ' ' 2026-03-20 16:26:25.810518 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.810522 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.810536 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.810540 | controller | available"' 2026-03-20 16:26:25.810544 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.810549 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.810553 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.810557 | controller | - ' exit 1' 2026-03-20 16:26:25.810561 | controller | - ' fi' 2026-03-20 16:26:25.810565 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.810570 | controller | - ' ' 2026-03-20 16:26:25.810574 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.810578 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.810582 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.810588 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.810593 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.810597 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.810601 | controller | -- \' 2026-03-20 16:26:25.810605 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.810609 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.810614 | controller | - ' logger.go:42: 16:16:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.810618 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.810624 | controller | - ' logger.go:42: 16:16:09 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.810628 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.810632 | controller | found' 2026-03-20 16:26:25.810637 | controller | - ' logger.go:42: 16:16:10 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.810641 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.810645 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.810650 | controller | - ' ' 2026-03-20 16:26:25.810654 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.810658 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.810662 | controller | - ' ' 2026-03-20 16:26:25.810666 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.810671 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.810678 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.810682 | controller | - ' exit 1' 2026-03-20 16:26:25.810687 | controller | - ' fi' 2026-03-20 16:26:25.810691 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.810710 | controller | - ' ' 2026-03-20 16:26:25.810714 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.810719 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.810723 | controller | | base64 -d)' 2026-03-20 16:26:25.810727 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.810731 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.810736 | controller | - ' exit 1' 2026-03-20 16:26:25.810740 | controller | - ' fi' 2026-03-20 16:26:25.810744 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.810748 | controller | - ' ' 2026-03-20 16:26:25.810752 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.810757 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.810761 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.810765 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.810769 | controller | --timeout=300s' 2026-03-20 16:26:25.810773 | controller | - ' ' 2026-03-20 16:26:25.810778 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.810782 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.810786 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.810790 | controller | - ' ' 2026-03-20 16:26:25.810794 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.810800 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.810805 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.810809 | controller | - ' ' 2026-03-20 16:26:25.810813 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.810817 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.810821 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.810834 | controller | available"' 2026-03-20 16:26:25.810838 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.810843 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.810847 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.810851 | controller | - ' exit 1' 2026-03-20 16:26:25.810855 | controller | - ' fi' 2026-03-20 16:26:25.810859 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.810864 | controller | - ' ' 2026-03-20 16:26:25.810868 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.810872 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.810876 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.810880 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.810884 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.810891 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.810895 | controller | -- \' 2026-03-20 16:26:25.810900 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.810904 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.810908 | controller | - ' logger.go:42: 16:16:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.810912 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.810916 | controller | - ' logger.go:42: 16:16:10 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.810920 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.810924 | controller | found' 2026-03-20 16:26:25.810929 | controller | - ' logger.go:42: 16:16:11 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.810933 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.810937 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.810941 | controller | - ' ' 2026-03-20 16:26:25.810945 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.810949 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.810954 | controller | - ' ' 2026-03-20 16:26:25.810958 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.810962 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.810966 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.810970 | controller | - ' exit 1' 2026-03-20 16:26:25.810974 | controller | - ' fi' 2026-03-20 16:26:25.810979 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.810983 | controller | - ' ' 2026-03-20 16:26:25.810987 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.810991 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.810995 | controller | | base64 -d)' 2026-03-20 16:26:25.810999 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.811004 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.811008 | controller | - ' exit 1' 2026-03-20 16:26:25.811012 | controller | - ' fi' 2026-03-20 16:26:25.811016 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.811020 | controller | - ' ' 2026-03-20 16:26:25.811024 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.811029 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.811033 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.811037 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.811041 | controller | --timeout=300s' 2026-03-20 16:26:25.811045 | controller | - ' ' 2026-03-20 16:26:25.811049 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.811054 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.811060 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.811066 | controller | - ' ' 2026-03-20 16:26:25.811071 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.811077 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.811085 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.811089 | controller | - ' ' 2026-03-20 16:26:25.811093 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.811097 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.811101 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.811106 | controller | available"' 2026-03-20 16:26:25.811118 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.811125 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.811129 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.811134 | controller | - ' exit 1' 2026-03-20 16:26:25.811138 | controller | - ' fi' 2026-03-20 16:26:25.811142 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.811146 | controller | - ' ' 2026-03-20 16:26:25.811150 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.811155 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.811159 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.811163 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.811167 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.811171 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.811175 | controller | -- \' 2026-03-20 16:26:25.811180 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.811186 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.811190 | controller | - ' logger.go:42: 16:16:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.811194 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.811198 | controller | - ' logger.go:42: 16:16:11 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.811202 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.811207 | controller | found' 2026-03-20 16:26:25.811213 | controller | - ' logger.go:42: 16:16:12 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.811218 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.811224 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.811230 | controller | - ' ' 2026-03-20 16:26:25.811236 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.811241 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.811247 | controller | - ' ' 2026-03-20 16:26:25.811252 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.811258 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.811264 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.811269 | controller | - ' exit 1' 2026-03-20 16:26:25.811275 | controller | - ' fi' 2026-03-20 16:26:25.811281 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.811287 | controller | - ' ' 2026-03-20 16:26:25.811292 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.811296 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.811303 | controller | | base64 -d)' 2026-03-20 16:26:25.811307 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.811311 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.811315 | controller | - ' exit 1' 2026-03-20 16:26:25.811319 | controller | - ' fi' 2026-03-20 16:26:25.811324 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.811330 | controller | - ' ' 2026-03-20 16:26:25.811334 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.811338 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.811343 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.811347 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.811351 | controller | --timeout=300s' 2026-03-20 16:26:25.811355 | controller | - ' ' 2026-03-20 16:26:25.811359 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.811363 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.811368 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.811372 | controller | - ' ' 2026-03-20 16:26:25.811376 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.811380 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.811384 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.811388 | controller | - ' ' 2026-03-20 16:26:25.811393 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.811397 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.811401 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.811405 | controller | available"' 2026-03-20 16:26:25.811409 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.811423 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.811427 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.811432 | controller | - ' exit 1' 2026-03-20 16:26:25.811436 | controller | - ' fi' 2026-03-20 16:26:25.811440 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.811444 | controller | - ' ' 2026-03-20 16:26:25.811448 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.811452 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.811456 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.811461 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.811465 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.811469 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.811473 | controller | -- \' 2026-03-20 16:26:25.811477 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.811482 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.811486 | controller | - ' logger.go:42: 16:16:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.811494 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.811499 | controller | - ' logger.go:42: 16:16:12 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.811504 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.811510 | controller | found' 2026-03-20 16:26:25.811516 | controller | - ' logger.go:42: 16:16:13 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.811522 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.811527 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.811533 | controller | - ' ' 2026-03-20 16:26:25.811539 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.811545 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.811550 | controller | - ' ' 2026-03-20 16:26:25.811554 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.811558 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.811562 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.811566 | controller | - ' exit 1' 2026-03-20 16:26:25.811571 | controller | - ' fi' 2026-03-20 16:26:25.811575 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.811579 | controller | - ' ' 2026-03-20 16:26:25.811583 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.811589 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.811595 | controller | | base64 -d)' 2026-03-20 16:26:25.811601 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.811607 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.811613 | controller | - ' exit 1' 2026-03-20 16:26:25.811622 | controller | - ' fi' 2026-03-20 16:26:25.811628 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.811634 | controller | - ' ' 2026-03-20 16:26:25.811640 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.811644 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.811649 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.811653 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.811657 | controller | --timeout=300s' 2026-03-20 16:26:25.811663 | controller | - ' ' 2026-03-20 16:26:25.811667 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.811671 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.811676 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.811680 | controller | - ' ' 2026-03-20 16:26:25.811684 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.811688 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.811709 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.811719 | controller | - ' ' 2026-03-20 16:26:25.811743 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.811748 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.811753 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.811759 | controller | available"' 2026-03-20 16:26:25.811768 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.811774 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.811791 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.811798 | controller | - ' exit 1' 2026-03-20 16:26:25.811804 | controller | - ' fi' 2026-03-20 16:26:25.811810 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.811816 | controller | - ' ' 2026-03-20 16:26:25.811822 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.811828 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.811834 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.811840 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.811846 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.811852 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.811858 | controller | -- \' 2026-03-20 16:26:25.811864 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.811870 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.811876 | controller | - ' logger.go:42: 16:16:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.811881 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.811891 | controller | - ' logger.go:42: 16:16:13 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.811897 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.811903 | controller | found' 2026-03-20 16:26:25.811909 | controller | - ' logger.go:42: 16:16:14 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.811915 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.811923 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.811929 | controller | - ' ' 2026-03-20 16:26:25.811935 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.811941 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.811947 | controller | - ' ' 2026-03-20 16:26:25.811952 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.811958 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.811964 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.811970 | controller | - ' exit 1' 2026-03-20 16:26:25.811976 | controller | - ' fi' 2026-03-20 16:26:25.811982 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.811988 | controller | - ' ' 2026-03-20 16:26:25.811993 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.811997 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.812001 | controller | | base64 -d)' 2026-03-20 16:26:25.812005 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.812009 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.812014 | controller | - ' exit 1' 2026-03-20 16:26:25.812018 | controller | - ' fi' 2026-03-20 16:26:25.812022 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.812026 | controller | - ' ' 2026-03-20 16:26:25.812030 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.812034 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.812055 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.812059 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.812063 | controller | --timeout=300s' 2026-03-20 16:26:25.812067 | controller | - ' ' 2026-03-20 16:26:25.812072 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.812076 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.812080 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.812084 | controller | - ' ' 2026-03-20 16:26:25.812088 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.812092 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.812097 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.812101 | controller | - ' ' 2026-03-20 16:26:25.812105 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.812109 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.812113 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.812117 | controller | available"' 2026-03-20 16:26:25.812122 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.812126 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.812151 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.812158 | controller | - ' exit 1' 2026-03-20 16:26:25.812164 | controller | - ' fi' 2026-03-20 16:26:25.812169 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.812175 | controller | - ' ' 2026-03-20 16:26:25.812181 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.812187 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.812192 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.812198 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.812204 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.812209 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.812215 | controller | -- \' 2026-03-20 16:26:25.812221 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.812226 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.812232 | controller | - ' logger.go:42: 16:16:14 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.812237 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.812243 | controller | - ' logger.go:42: 16:16:14 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.812249 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.812255 | controller | found' 2026-03-20 16:26:25.812265 | controller | - ' logger.go:42: 16:16:15 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.812270 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.812276 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.812287 | controller | - ' ' 2026-03-20 16:26:25.812294 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.812300 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.812305 | controller | - ' ' 2026-03-20 16:26:25.812311 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.812317 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.812324 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.812329 | controller | - ' exit 1' 2026-03-20 16:26:25.812335 | controller | - ' fi' 2026-03-20 16:26:25.812341 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.812347 | controller | - ' ' 2026-03-20 16:26:25.812353 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.812358 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.812364 | controller | | base64 -d)' 2026-03-20 16:26:25.812370 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.812376 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.812382 | controller | - ' exit 1' 2026-03-20 16:26:25.812388 | controller | - ' fi' 2026-03-20 16:26:25.812393 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.812398 | controller | - ' ' 2026-03-20 16:26:25.812404 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.812410 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.812416 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.812422 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.812428 | controller | --timeout=300s' 2026-03-20 16:26:25.812433 | controller | - ' ' 2026-03-20 16:26:25.812439 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.812445 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.812451 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.812457 | controller | - ' ' 2026-03-20 16:26:25.812463 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.812469 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.812474 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.812480 | controller | - ' ' 2026-03-20 16:26:25.812486 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.812492 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.812498 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.812504 | controller | available"' 2026-03-20 16:26:25.812509 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.812515 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.812521 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.812527 | controller | - ' exit 1' 2026-03-20 16:26:25.812546 | controller | - ' fi' 2026-03-20 16:26:25.812555 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.812561 | controller | - ' ' 2026-03-20 16:26:25.812570 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.812577 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.812584 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.812590 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.812595 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.812601 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.812607 | controller | -- \' 2026-03-20 16:26:25.812613 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.812619 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.812625 | controller | - ' logger.go:42: 16:16:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.812631 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.812637 | controller | - ' logger.go:42: 16:16:16 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.812656 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.812663 | controller | found' 2026-03-20 16:26:25.812669 | controller | - ' logger.go:42: 16:16:17 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.812674 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.812680 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.812686 | controller | - ' ' 2026-03-20 16:26:25.812708 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.812720 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.812726 | controller | - ' ' 2026-03-20 16:26:25.812732 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.812737 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.812743 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.812749 | controller | - ' exit 1' 2026-03-20 16:26:25.812755 | controller | - ' fi' 2026-03-20 16:26:25.812761 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.812767 | controller | - ' ' 2026-03-20 16:26:25.812777 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.812783 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.812789 | controller | | base64 -d)' 2026-03-20 16:26:25.812795 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.812801 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.812806 | controller | - ' exit 1' 2026-03-20 16:26:25.812812 | controller | - ' fi' 2026-03-20 16:26:25.812818 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.812824 | controller | - ' ' 2026-03-20 16:26:25.812829 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.812835 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.812841 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.812847 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.812853 | controller | --timeout=300s' 2026-03-20 16:26:25.812859 | controller | - ' ' 2026-03-20 16:26:25.812865 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.812871 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.812880 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.812886 | controller | - ' ' 2026-03-20 16:26:25.812891 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.812897 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.812903 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.812909 | controller | - ' ' 2026-03-20 16:26:25.812915 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.812921 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.812926 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.812932 | controller | available"' 2026-03-20 16:26:25.812938 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.812944 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.812949 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.812955 | controller | - ' exit 1' 2026-03-20 16:26:25.812961 | controller | - ' fi' 2026-03-20 16:26:25.812978 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.812985 | controller | - ' ' 2026-03-20 16:26:25.812990 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.812996 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.813002 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.813007 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.813013 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.813019 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.813025 | controller | -- \' 2026-03-20 16:26:25.813031 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.813036 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.813042 | controller | - ' logger.go:42: 16:16:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.813048 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.813054 | controller | - ' logger.go:42: 16:16:17 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.813059 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.813065 | controller | found' 2026-03-20 16:26:25.813071 | controller | - ' logger.go:42: 16:16:18 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.813076 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.813082 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.813087 | controller | - ' ' 2026-03-20 16:26:25.813093 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.813099 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.813105 | controller | - ' ' 2026-03-20 16:26:25.813114 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.813120 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.813126 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.813136 | controller | - ' exit 1' 2026-03-20 16:26:25.813142 | controller | - ' fi' 2026-03-20 16:26:25.813148 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.813154 | controller | - ' ' 2026-03-20 16:26:25.813160 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.813165 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.813171 | controller | | base64 -d)' 2026-03-20 16:26:25.813177 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.813183 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.813187 | controller | - ' exit 1' 2026-03-20 16:26:25.813191 | controller | - ' fi' 2026-03-20 16:26:25.813196 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.813200 | controller | - ' ' 2026-03-20 16:26:25.813204 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.813208 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.813212 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.813216 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.813220 | controller | --timeout=300s' 2026-03-20 16:26:25.813225 | controller | - ' ' 2026-03-20 16:26:25.813229 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.813235 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.813239 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.813243 | controller | - ' ' 2026-03-20 16:26:25.813247 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.813252 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.813258 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.813262 | controller | - ' ' 2026-03-20 16:26:25.813266 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.813270 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.813274 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.813278 | controller | available"' 2026-03-20 16:26:25.813283 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.813287 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.813291 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.813295 | controller | - ' exit 1' 2026-03-20 16:26:25.813299 | controller | - ' fi' 2026-03-20 16:26:25.813303 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.813317 | controller | - ' ' 2026-03-20 16:26:25.813322 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.813326 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.813330 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.813334 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.813339 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.813345 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.813349 | controller | -- \' 2026-03-20 16:26:25.813353 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.813358 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.813362 | controller | - ' logger.go:42: 16:16:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.813366 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.813370 | controller | - ' logger.go:42: 16:16:18 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.813374 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.813378 | controller | found' 2026-03-20 16:26:25.813384 | controller | - ' logger.go:42: 16:16:19 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.813389 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.813393 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.813397 | controller | - ' ' 2026-03-20 16:26:25.813401 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.813405 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.813409 | controller | - ' ' 2026-03-20 16:26:25.813414 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.813419 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.813425 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.813431 | controller | - ' exit 1' 2026-03-20 16:26:25.813437 | controller | - ' fi' 2026-03-20 16:26:25.813442 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.813448 | controller | - ' ' 2026-03-20 16:26:25.813454 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.813460 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.813465 | controller | | base64 -d)' 2026-03-20 16:26:25.813471 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.813477 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.813482 | controller | - ' exit 1' 2026-03-20 16:26:25.813486 | controller | - ' fi' 2026-03-20 16:26:25.813491 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.813495 | controller | - ' ' 2026-03-20 16:26:25.813499 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.813503 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.813507 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.813511 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.813515 | controller | --timeout=300s' 2026-03-20 16:26:25.813520 | controller | - ' ' 2026-03-20 16:26:25.813524 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.813528 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.813532 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.813536 | controller | - ' ' 2026-03-20 16:26:25.813540 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.813545 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.813549 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.813556 | controller | - ' ' 2026-03-20 16:26:25.813560 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.813564 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.813568 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.813572 | controller | available"' 2026-03-20 16:26:25.813576 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.813580 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.813584 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.813589 | controller | - ' exit 1' 2026-03-20 16:26:25.813593 | controller | - ' fi' 2026-03-20 16:26:25.813597 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.813601 | controller | - ' ' 2026-03-20 16:26:25.813615 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.813619 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.813623 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.813627 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.813632 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.813636 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.813640 | controller | -- \' 2026-03-20 16:26:25.813644 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.813648 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.813653 | controller | - ' logger.go:42: 16:16:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.813657 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.813661 | controller | - ' logger.go:42: 16:16:19 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.813666 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.813671 | controller | found' 2026-03-20 16:26:25.813677 | controller | - ' logger.go:42: 16:16:20 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.813683 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.813689 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.813710 | controller | - ' ' 2026-03-20 16:26:25.813715 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.813719 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.813723 | controller | - ' ' 2026-03-20 16:26:25.813727 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.813731 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.813735 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.813740 | controller | - ' exit 1' 2026-03-20 16:26:25.813744 | controller | - ' fi' 2026-03-20 16:26:25.813748 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.813752 | controller | - ' ' 2026-03-20 16:26:25.813756 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.813761 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.813765 | controller | | base64 -d)' 2026-03-20 16:26:25.813769 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.813776 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.813780 | controller | - ' exit 1' 2026-03-20 16:26:25.813784 | controller | - ' fi' 2026-03-20 16:26:25.813789 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.813793 | controller | - ' ' 2026-03-20 16:26:25.813797 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.813801 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.813805 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.813809 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.813813 | controller | --timeout=300s' 2026-03-20 16:26:25.813818 | controller | - ' ' 2026-03-20 16:26:25.813822 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.813826 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.813830 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.813834 | controller | - ' ' 2026-03-20 16:26:25.813838 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.813842 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.813847 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.813851 | controller | - ' ' 2026-03-20 16:26:25.813855 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.813859 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.813863 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.813867 | controller | available"' 2026-03-20 16:26:25.813872 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.813876 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.813880 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.813884 | controller | - ' exit 1' 2026-03-20 16:26:25.813888 | controller | - ' fi' 2026-03-20 16:26:25.813892 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.813897 | controller | - ' ' 2026-03-20 16:26:25.813901 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.813914 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.813919 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.813923 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.813928 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.813932 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.813936 | controller | -- \' 2026-03-20 16:26:25.813940 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.813944 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.813949 | controller | - ' logger.go:42: 16:16:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.813955 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.813959 | controller | - ' logger.go:42: 16:16:20 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.813963 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.813967 | controller | found' 2026-03-20 16:26:25.813971 | controller | - ' logger.go:42: 16:16:21 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.813976 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.813980 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.813984 | controller | - ' ' 2026-03-20 16:26:25.813993 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.813997 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.814001 | controller | - ' ' 2026-03-20 16:26:25.814005 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.814009 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.814013 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.814018 | controller | - ' exit 1' 2026-03-20 16:26:25.814022 | controller | - ' fi' 2026-03-20 16:26:25.814026 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.814030 | controller | - ' ' 2026-03-20 16:26:25.814034 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.814039 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.814043 | controller | | base64 -d)' 2026-03-20 16:26:25.814047 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.814051 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.814055 | controller | - ' exit 1' 2026-03-20 16:26:25.814059 | controller | - ' fi' 2026-03-20 16:26:25.814064 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.814068 | controller | - ' ' 2026-03-20 16:26:25.814072 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.814076 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.814080 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.814084 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.814088 | controller | --timeout=300s' 2026-03-20 16:26:25.814093 | controller | - ' ' 2026-03-20 16:26:25.814097 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.814101 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.814105 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.814109 | controller | - ' ' 2026-03-20 16:26:25.814114 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.814118 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.814122 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.814126 | controller | - ' ' 2026-03-20 16:26:25.814130 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.814134 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.814139 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.814145 | controller | available"' 2026-03-20 16:26:25.814154 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.814162 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.814170 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.814176 | controller | - ' exit 1' 2026-03-20 16:26:25.814182 | controller | - ' fi' 2026-03-20 16:26:25.814188 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.814193 | controller | - ' ' 2026-03-20 16:26:25.814197 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.814211 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.814215 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.814219 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.814224 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.814228 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.814232 | controller | -- \' 2026-03-20 16:26:25.814236 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.814240 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.814248 | controller | - ' logger.go:42: 16:16:21 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.814252 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.814256 | controller | - ' logger.go:42: 16:16:21 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.814260 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.814265 | controller | found' 2026-03-20 16:26:25.814269 | controller | - ' logger.go:42: 16:16:22 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.814273 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.814277 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.814281 | controller | - ' ' 2026-03-20 16:26:25.814285 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.814290 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.814294 | controller | - ' ' 2026-03-20 16:26:25.814299 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.814304 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.814309 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.814313 | controller | - ' exit 1' 2026-03-20 16:26:25.814318 | controller | - ' fi' 2026-03-20 16:26:25.814323 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.814328 | controller | - ' ' 2026-03-20 16:26:25.814332 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.814336 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.814340 | controller | | base64 -d)' 2026-03-20 16:26:25.814345 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.814349 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.814353 | controller | - ' exit 1' 2026-03-20 16:26:25.814357 | controller | - ' fi' 2026-03-20 16:26:25.814361 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.814365 | controller | - ' ' 2026-03-20 16:26:25.814369 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.814374 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.814380 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.814384 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.814389 | controller | --timeout=300s' 2026-03-20 16:26:25.814393 | controller | - ' ' 2026-03-20 16:26:25.814397 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.814401 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.814405 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.814409 | controller | - ' ' 2026-03-20 16:26:25.814414 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.814418 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.814422 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.814426 | controller | - ' ' 2026-03-20 16:26:25.814430 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.814434 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.814441 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.814445 | controller | available"' 2026-03-20 16:26:25.814449 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.814453 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.814457 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.814461 | controller | - ' exit 1' 2026-03-20 16:26:25.814465 | controller | - ' fi' 2026-03-20 16:26:25.814470 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.814474 | controller | - ' ' 2026-03-20 16:26:25.814478 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.814482 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.814494 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.814499 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.814503 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.814507 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.814511 | controller | -- \' 2026-03-20 16:26:25.814516 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.814520 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.814524 | controller | - ' logger.go:42: 16:16:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.814528 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.814532 | controller | - ' logger.go:42: 16:16:22 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.814537 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.814541 | controller | found' 2026-03-20 16:26:25.814545 | controller | - ' logger.go:42: 16:16:23 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.814549 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.814553 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.814560 | controller | - ' ' 2026-03-20 16:26:25.814566 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.814570 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.814576 | controller | - ' ' 2026-03-20 16:26:25.814580 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.814584 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.814589 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.814593 | controller | - ' exit 1' 2026-03-20 16:26:25.814597 | controller | - ' fi' 2026-03-20 16:26:25.814601 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.814605 | controller | - ' ' 2026-03-20 16:26:25.814609 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.814614 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.814618 | controller | | base64 -d)' 2026-03-20 16:26:25.814622 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.814626 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.814630 | controller | - ' exit 1' 2026-03-20 16:26:25.814635 | controller | - ' fi' 2026-03-20 16:26:25.814641 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.814647 | controller | - ' ' 2026-03-20 16:26:25.814653 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.814659 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.814664 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.814668 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.814672 | controller | --timeout=300s' 2026-03-20 16:26:25.814676 | controller | - ' ' 2026-03-20 16:26:25.814680 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.814684 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.814688 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.814708 | controller | - ' ' 2026-03-20 16:26:25.814719 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.814725 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.814730 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.814734 | controller | - ' ' 2026-03-20 16:26:25.814738 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.814743 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.814747 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.814751 | controller | available"' 2026-03-20 16:26:25.814755 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.814759 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.814763 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.814768 | controller | - ' exit 1' 2026-03-20 16:26:25.814772 | controller | - ' fi' 2026-03-20 16:26:25.814776 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.814780 | controller | - ' ' 2026-03-20 16:26:25.814784 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.814791 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.814805 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.814810 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.814814 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.814818 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.814823 | controller | -- \' 2026-03-20 16:26:25.814827 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.814831 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.814835 | controller | - ' logger.go:42: 16:16:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.814840 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.814844 | controller | - ' logger.go:42: 16:16:24 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.814848 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.814852 | controller | found' 2026-03-20 16:26:25.814856 | controller | - ' logger.go:42: 16:16:25 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.814861 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.814866 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.814872 | controller | - ' ' 2026-03-20 16:26:25.814878 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.814883 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.814889 | controller | - ' ' 2026-03-20 16:26:25.814894 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.814900 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.814906 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.814911 | controller | - ' exit 1' 2026-03-20 16:26:25.814917 | controller | - ' fi' 2026-03-20 16:26:25.814923 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.814928 | controller | - ' ' 2026-03-20 16:26:25.814934 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.814940 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.814946 | controller | | base64 -d)' 2026-03-20 16:26:25.814951 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.814957 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.814961 | controller | - ' exit 1' 2026-03-20 16:26:25.814965 | controller | - ' fi' 2026-03-20 16:26:25.814970 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.814974 | controller | - ' ' 2026-03-20 16:26:25.814978 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.814982 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.814986 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.814991 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.814995 | controller | --timeout=300s' 2026-03-20 16:26:25.814999 | controller | - ' ' 2026-03-20 16:26:25.815003 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.815007 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.815026 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.815033 | controller | - ' ' 2026-03-20 16:26:25.815039 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.815044 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.815050 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.815056 | controller | - ' ' 2026-03-20 16:26:25.815062 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.815067 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.815073 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.815079 | controller | available"' 2026-03-20 16:26:25.815085 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.815091 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.815097 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.815102 | controller | - ' exit 1' 2026-03-20 16:26:25.815108 | controller | - ' fi' 2026-03-20 16:26:25.815114 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.815120 | controller | - ' ' 2026-03-20 16:26:25.815126 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.815131 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.815136 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.815150 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.815155 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.815160 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.815164 | controller | -- \' 2026-03-20 16:26:25.815168 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.815173 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.815177 | controller | - ' logger.go:42: 16:16:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.815181 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.815185 | controller | - ' logger.go:42: 16:16:25 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.815189 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.815194 | controller | found' 2026-03-20 16:26:25.815198 | controller | - ' logger.go:42: 16:16:26 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.815202 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.815206 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.815210 | controller | - ' ' 2026-03-20 16:26:25.815215 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.815219 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.815223 | controller | - ' ' 2026-03-20 16:26:25.815230 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.815235 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.815239 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.815247 | controller | - ' exit 1' 2026-03-20 16:26:25.815253 | controller | - ' fi' 2026-03-20 16:26:25.815259 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.815265 | controller | - ' ' 2026-03-20 16:26:25.815270 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.815276 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.815282 | controller | | base64 -d)' 2026-03-20 16:26:25.815288 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.815293 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.815299 | controller | - ' exit 1' 2026-03-20 16:26:25.815305 | controller | - ' fi' 2026-03-20 16:26:25.815310 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.815316 | controller | - ' ' 2026-03-20 16:26:25.815325 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.815331 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.815337 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.815342 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.815348 | controller | --timeout=300s' 2026-03-20 16:26:25.815353 | controller | - ' ' 2026-03-20 16:26:25.815357 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.815362 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.815366 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.815370 | controller | - ' ' 2026-03-20 16:26:25.815374 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.815378 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.815382 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.815386 | controller | - ' ' 2026-03-20 16:26:25.815391 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.815399 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.815403 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.815407 | controller | available"' 2026-03-20 16:26:25.815411 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.815415 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.815419 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.815424 | controller | - ' exit 1' 2026-03-20 16:26:25.815428 | controller | - ' fi' 2026-03-20 16:26:25.815432 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.815436 | controller | - ' ' 2026-03-20 16:26:25.815440 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.815445 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.815449 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.815462 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.815466 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.815471 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.815478 | controller | -- \' 2026-03-20 16:26:25.815482 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.815487 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.815491 | controller | - ' logger.go:42: 16:16:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.815495 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.815499 | controller | - ' logger.go:42: 16:16:26 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.815503 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.815507 | controller | found' 2026-03-20 16:26:25.815512 | controller | - ' logger.go:42: 16:16:27 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.815516 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.815520 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.815524 | controller | - ' ' 2026-03-20 16:26:25.815528 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.815533 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.815537 | controller | - ' ' 2026-03-20 16:26:25.815541 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.815545 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.815549 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.815554 | controller | - ' exit 1' 2026-03-20 16:26:25.815558 | controller | - ' fi' 2026-03-20 16:26:25.815563 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.815568 | controller | - ' ' 2026-03-20 16:26:25.815577 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.815583 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.815588 | controller | | base64 -d)' 2026-03-20 16:26:25.815592 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.815597 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.815601 | controller | - ' exit 1' 2026-03-20 16:26:25.815605 | controller | - ' fi' 2026-03-20 16:26:25.815609 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.815613 | controller | - ' ' 2026-03-20 16:26:25.815618 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.815622 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.815626 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.815630 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.815634 | controller | --timeout=300s' 2026-03-20 16:26:25.815639 | controller | - ' ' 2026-03-20 16:26:25.815643 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.815647 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.815651 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.815655 | controller | - ' ' 2026-03-20 16:26:25.815660 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.815664 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.815668 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.815674 | controller | - ' ' 2026-03-20 16:26:25.815678 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.815682 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.815687 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.815704 | controller | available"' 2026-03-20 16:26:25.815712 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.815716 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.815722 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.815750 | controller | - ' exit 1' 2026-03-20 16:26:25.815757 | controller | - ' fi' 2026-03-20 16:26:25.815766 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.815772 | controller | - ' ' 2026-03-20 16:26:25.815778 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.815784 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.815790 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.815795 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.815809 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.815814 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.815819 | controller | -- \' 2026-03-20 16:26:25.815823 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.815827 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.815832 | controller | - ' logger.go:42: 16:16:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.815836 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.815842 | controller | - ' logger.go:42: 16:16:27 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.815848 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.815854 | controller | found' 2026-03-20 16:26:25.815860 | controller | - ' logger.go:42: 16:16:28 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.815866 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.815872 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.815878 | controller | - ' ' 2026-03-20 16:26:25.815884 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.815889 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.815895 | controller | - ' ' 2026-03-20 16:26:25.815900 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.815905 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.815909 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.815913 | controller | - ' exit 1' 2026-03-20 16:26:25.815917 | controller | - ' fi' 2026-03-20 16:26:25.815921 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.815929 | controller | - ' ' 2026-03-20 16:26:25.815933 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.815939 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.815945 | controller | | base64 -d)' 2026-03-20 16:26:25.815951 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.815961 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.815967 | controller | - ' exit 1' 2026-03-20 16:26:25.815973 | controller | - ' fi' 2026-03-20 16:26:25.815979 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.815984 | controller | - ' ' 2026-03-20 16:26:25.815990 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.815996 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.816002 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.816007 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.816013 | controller | --timeout=300s' 2026-03-20 16:26:25.816018 | controller | - ' ' 2026-03-20 16:26:25.816025 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.816029 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.816035 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.816041 | controller | - ' ' 2026-03-20 16:26:25.816047 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.816053 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.816059 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.816065 | controller | - ' ' 2026-03-20 16:26:25.816070 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.816076 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.816082 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.816088 | controller | available"' 2026-03-20 16:26:25.816093 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.816099 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.816105 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.816111 | controller | - ' exit 1' 2026-03-20 16:26:25.816116 | controller | - ' fi' 2026-03-20 16:26:25.816122 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.816128 | controller | - ' ' 2026-03-20 16:26:25.816134 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.816140 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.816145 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.816151 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.816168 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.816174 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.816180 | controller | -- \' 2026-03-20 16:26:25.816186 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.816192 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.816198 | controller | - ' logger.go:42: 16:16:28 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.816203 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.816213 | controller | - ' logger.go:42: 16:16:28 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.816219 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.816224 | controller | found' 2026-03-20 16:26:25.816230 | controller | - ' logger.go:42: 16:16:29 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.816236 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.816242 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.816248 | controller | - ' ' 2026-03-20 16:26:25.816254 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.816260 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.816266 | controller | - ' ' 2026-03-20 16:26:25.816271 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.816277 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.816283 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.816289 | controller | - ' exit 1' 2026-03-20 16:26:25.816294 | controller | - ' fi' 2026-03-20 16:26:25.816300 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.816306 | controller | - ' ' 2026-03-20 16:26:25.816312 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.816318 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.816324 | controller | | base64 -d)' 2026-03-20 16:26:25.816330 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.816335 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.816341 | controller | - ' exit 1' 2026-03-20 16:26:25.816347 | controller | - ' fi' 2026-03-20 16:26:25.816353 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.816359 | controller | - ' ' 2026-03-20 16:26:25.816365 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.816371 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.816376 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.816382 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.816388 | controller | --timeout=300s' 2026-03-20 16:26:25.816397 | controller | - ' ' 2026-03-20 16:26:25.816403 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.816409 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.816414 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.816420 | controller | - ' ' 2026-03-20 16:26:25.816426 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.816431 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.816437 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.816443 | controller | - ' ' 2026-03-20 16:26:25.816449 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.816455 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.816460 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.816466 | controller | available"' 2026-03-20 16:26:25.816472 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.816478 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.816486 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.816492 | controller | - ' exit 1' 2026-03-20 16:26:25.816498 | controller | - ' fi' 2026-03-20 16:26:25.816504 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.816510 | controller | - ' ' 2026-03-20 16:26:25.816516 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.816521 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.816527 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.816533 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.816538 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.816555 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.816561 | controller | -- \' 2026-03-20 16:26:25.816567 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.816573 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.816578 | controller | - ' logger.go:42: 16:16:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.816584 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.816590 | controller | - ' logger.go:42: 16:16:29 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.816596 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.816602 | controller | found' 2026-03-20 16:26:25.816608 | controller | - ' logger.go:42: 16:16:30 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.816613 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.816619 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.816625 | controller | - ' ' 2026-03-20 16:26:25.816631 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.816637 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.816643 | controller | - ' ' 2026-03-20 16:26:25.816648 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.816654 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.816660 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.816667 | controller | - ' exit 1' 2026-03-20 16:26:25.816673 | controller | - ' fi' 2026-03-20 16:26:25.816683 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.816689 | controller | - ' ' 2026-03-20 16:26:25.816713 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.816723 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.816729 | controller | | base64 -d)' 2026-03-20 16:26:25.816734 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.816740 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.816746 | controller | - ' exit 1' 2026-03-20 16:26:25.816752 | controller | - ' fi' 2026-03-20 16:26:25.816757 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.816763 | controller | - ' ' 2026-03-20 16:26:25.816769 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.816775 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.816784 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.816794 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.816800 | controller | --timeout=300s' 2026-03-20 16:26:25.816806 | controller | - ' ' 2026-03-20 16:26:25.816811 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.816817 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.816826 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.816831 | controller | - ' ' 2026-03-20 16:26:25.816837 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.816843 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.816849 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.816854 | controller | - ' ' 2026-03-20 16:26:25.816860 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.816866 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.816872 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.816878 | controller | available"' 2026-03-20 16:26:25.816884 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.816890 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.816895 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.816901 | controller | - ' exit 1' 2026-03-20 16:26:25.816907 | controller | - ' fi' 2026-03-20 16:26:25.816912 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.816918 | controller | - ' ' 2026-03-20 16:26:25.816924 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.816929 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.816935 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.816941 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.816946 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.816964 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.816972 | controller | -- \' 2026-03-20 16:26:25.816978 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.816983 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.816989 | controller | - ' logger.go:42: 16:16:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.816995 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.817001 | controller | - ' logger.go:42: 16:16:31 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.817007 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.817012 | controller | found' 2026-03-20 16:26:25.817018 | controller | - ' logger.go:42: 16:16:32 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.817024 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.817030 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.817035 | controller | - ' ' 2026-03-20 16:26:25.817041 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.817052 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.817058 | controller | - ' ' 2026-03-20 16:26:25.817064 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.817070 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.817076 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.817082 | controller | - ' exit 1' 2026-03-20 16:26:25.817087 | controller | - ' fi' 2026-03-20 16:26:25.817093 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.817099 | controller | - ' ' 2026-03-20 16:26:25.817105 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.817110 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.817116 | controller | | base64 -d)' 2026-03-20 16:26:25.817122 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.817128 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.817134 | controller | - ' exit 1' 2026-03-20 16:26:25.817140 | controller | - ' fi' 2026-03-20 16:26:25.817145 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.817151 | controller | - ' ' 2026-03-20 16:26:25.817157 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.817163 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.817168 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.817174 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.817180 | controller | --timeout=300s' 2026-03-20 16:26:25.817186 | controller | - ' ' 2026-03-20 16:26:25.817191 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.817197 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.817203 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.817209 | controller | - ' ' 2026-03-20 16:26:25.817214 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.817220 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.817226 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.817232 | controller | - ' ' 2026-03-20 16:26:25.817238 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.817244 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.817250 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.817255 | controller | available"' 2026-03-20 16:26:25.817261 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.817270 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.817276 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.817282 | controller | - ' exit 1' 2026-03-20 16:26:25.817287 | controller | - ' fi' 2026-03-20 16:26:25.817293 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.817299 | controller | - ' ' 2026-03-20 16:26:25.817305 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.817311 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.817320 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.817326 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.817331 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.817337 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.817343 | controller | -- \' 2026-03-20 16:26:25.817360 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.817366 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.817372 | controller | - ' logger.go:42: 16:16:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.817378 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.817384 | controller | - ' logger.go:42: 16:16:32 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.817390 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.817395 | controller | found' 2026-03-20 16:26:25.817401 | controller | - ' logger.go:42: 16:16:33 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.817407 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.817413 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.817419 | controller | - ' ' 2026-03-20 16:26:25.817425 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.817430 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.817436 | controller | - ' ' 2026-03-20 16:26:25.817442 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.817448 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.817453 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.817459 | controller | - ' exit 1' 2026-03-20 16:26:25.817465 | controller | - ' fi' 2026-03-20 16:26:25.817471 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.817476 | controller | - ' ' 2026-03-20 16:26:25.817482 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.817488 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.817494 | controller | | base64 -d)' 2026-03-20 16:26:25.817500 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.817505 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.817511 | controller | - ' exit 1' 2026-03-20 16:26:25.817517 | controller | - ' fi' 2026-03-20 16:26:25.817523 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.817528 | controller | - ' ' 2026-03-20 16:26:25.817534 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.817539 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.817545 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.817554 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.817560 | controller | --timeout=300s' 2026-03-20 16:26:25.817566 | controller | - ' ' 2026-03-20 16:26:25.817572 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.817577 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.817583 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.817592 | controller | - ' ' 2026-03-20 16:26:25.817598 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.817604 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.817610 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.817616 | controller | - ' ' 2026-03-20 16:26:25.817621 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.817627 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.817633 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.817639 | controller | available"' 2026-03-20 16:26:25.817644 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.817650 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.817656 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.817662 | controller | - ' exit 1' 2026-03-20 16:26:25.817667 | controller | - ' fi' 2026-03-20 16:26:25.817673 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.817679 | controller | - ' ' 2026-03-20 16:26:25.817685 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.817711 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.817721 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.817727 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.817733 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.817738 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.817744 | controller | -- \' 2026-03-20 16:26:25.817759 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.817766 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.817772 | controller | - ' logger.go:42: 16:16:33 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.817778 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.817784 | controller | - ' logger.go:42: 16:16:33 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.817790 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.817796 | controller | found' 2026-03-20 16:26:25.817802 | controller | - ' logger.go:42: 16:16:34 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.817807 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.817813 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.817819 | controller | - ' ' 2026-03-20 16:26:25.817825 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.817830 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.817836 | controller | - ' ' 2026-03-20 16:26:25.817842 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.817848 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.817854 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.817863 | controller | - ' exit 1' 2026-03-20 16:26:25.817869 | controller | - ' fi' 2026-03-20 16:26:25.817878 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.817884 | controller | - ' ' 2026-03-20 16:26:25.817889 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.817895 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.817901 | controller | | base64 -d)' 2026-03-20 16:26:25.817907 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.817913 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.817918 | controller | - ' exit 1' 2026-03-20 16:26:25.817924 | controller | - ' fi' 2026-03-20 16:26:25.817930 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.817936 | controller | - ' ' 2026-03-20 16:26:25.817942 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.817948 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.817954 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.817959 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.817965 | controller | --timeout=300s' 2026-03-20 16:26:25.817971 | controller | - ' ' 2026-03-20 16:26:25.817977 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.817982 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.817988 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.817994 | controller | - ' ' 2026-03-20 16:26:25.818000 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.818006 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.818011 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.818017 | controller | - ' ' 2026-03-20 16:26:25.818023 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.818029 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.818035 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.818040 | controller | available"' 2026-03-20 16:26:25.818046 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.818052 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.818057 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.818063 | controller | - ' exit 1' 2026-03-20 16:26:25.818069 | controller | - ' fi' 2026-03-20 16:26:25.818078 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.818084 | controller | - ' ' 2026-03-20 16:26:25.818089 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.818095 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.818101 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.818107 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.818115 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.818121 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.818130 | controller | -- \' 2026-03-20 16:26:25.818136 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.818152 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.818158 | controller | - ' logger.go:42: 16:16:34 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.818164 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.818170 | controller | - ' logger.go:42: 16:16:34 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.818176 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.818182 | controller | found' 2026-03-20 16:26:25.818187 | controller | - ' logger.go:42: 16:16:35 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.818193 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.818199 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.818205 | controller | - ' ' 2026-03-20 16:26:25.818211 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.818217 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.818223 | controller | - ' ' 2026-03-20 16:26:25.818228 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.818234 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.818240 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.818246 | controller | - ' exit 1' 2026-03-20 16:26:25.818252 | controller | - ' fi' 2026-03-20 16:26:25.818257 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.818263 | controller | - ' ' 2026-03-20 16:26:25.818269 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.818275 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.818281 | controller | | base64 -d)' 2026-03-20 16:26:25.818286 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.818292 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.818298 | controller | - ' exit 1' 2026-03-20 16:26:25.818304 | controller | - ' fi' 2026-03-20 16:26:25.818310 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.818316 | controller | - ' ' 2026-03-20 16:26:25.818322 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.818327 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.818333 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.818339 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.818345 | controller | --timeout=300s' 2026-03-20 16:26:25.818350 | controller | - ' ' 2026-03-20 16:26:25.818356 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.818362 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.818368 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.818374 | controller | - ' ' 2026-03-20 16:26:25.818379 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.818385 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.818391 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.818397 | controller | - ' ' 2026-03-20 16:26:25.818408 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.818415 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.818420 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.818426 | controller | available"' 2026-03-20 16:26:25.818432 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.818438 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.818444 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.818449 | controller | - ' exit 1' 2026-03-20 16:26:25.818455 | controller | - ' fi' 2026-03-20 16:26:25.818461 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.818467 | controller | - ' ' 2026-03-20 16:26:25.818473 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.818479 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.818485 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.818490 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.818496 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.818502 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.818508 | controller | -- \' 2026-03-20 16:26:25.818514 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.818530 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.818536 | controller | - ' logger.go:42: 16:16:35 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.818542 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.818548 | controller | - ' logger.go:42: 16:16:35 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.818554 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.818559 | controller | found' 2026-03-20 16:26:25.818565 | controller | - ' logger.go:42: 16:16:36 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.818571 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.818577 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.818583 | controller | - ' ' 2026-03-20 16:26:25.818589 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.818594 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.818600 | controller | - ' ' 2026-03-20 16:26:25.818606 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.818612 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.818617 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.818623 | controller | - ' exit 1' 2026-03-20 16:26:25.818629 | controller | - ' fi' 2026-03-20 16:26:25.818635 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.818641 | controller | - ' ' 2026-03-20 16:26:25.818646 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.818652 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.818658 | controller | | base64 -d)' 2026-03-20 16:26:25.818664 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.818670 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.818679 | controller | - ' exit 1' 2026-03-20 16:26:25.818685 | controller | - ' fi' 2026-03-20 16:26:25.818706 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.818715 | controller | - ' ' 2026-03-20 16:26:25.818721 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.818727 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.818732 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.818738 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.818744 | controller | --timeout=300s' 2026-03-20 16:26:25.818750 | controller | - ' ' 2026-03-20 16:26:25.818756 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.818762 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.818768 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.818773 | controller | - ' ' 2026-03-20 16:26:25.818779 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.818785 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.818791 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.818796 | controller | - ' ' 2026-03-20 16:26:25.818802 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.818808 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.818814 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.818819 | controller | available"' 2026-03-20 16:26:25.818825 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.818831 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.818837 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.818843 | controller | - ' exit 1' 2026-03-20 16:26:25.818849 | controller | - ' fi' 2026-03-20 16:26:25.818855 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.818860 | controller | - ' ' 2026-03-20 16:26:25.818866 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.818872 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.818878 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.818883 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.818889 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.818895 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.818900 | controller | -- \' 2026-03-20 16:26:25.818906 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.818912 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.818928 | controller | - ' logger.go:42: 16:16:36 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.818935 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.818941 | controller | - ' logger.go:42: 16:16:36 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.818950 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.818956 | controller | found' 2026-03-20 16:26:25.818965 | controller | - ' logger.go:42: 16:16:37 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.818971 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.818976 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.818982 | controller | - ' ' 2026-03-20 16:26:25.818988 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.818993 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.818999 | controller | - ' ' 2026-03-20 16:26:25.819005 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.819011 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.819017 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.819023 | controller | - ' exit 1' 2026-03-20 16:26:25.819029 | controller | - ' fi' 2026-03-20 16:26:25.819035 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.819040 | controller | - ' ' 2026-03-20 16:26:25.819062 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.819068 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.819074 | controller | | base64 -d)' 2026-03-20 16:26:25.819080 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.819085 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.819091 | controller | - ' exit 1' 2026-03-20 16:26:25.819097 | controller | - ' fi' 2026-03-20 16:26:25.819103 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.819109 | controller | - ' ' 2026-03-20 16:26:25.819115 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.819121 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.819126 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.819132 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.819138 | controller | --timeout=300s' 2026-03-20 16:26:25.819144 | controller | - ' ' 2026-03-20 16:26:25.819150 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.819155 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.819161 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.819167 | controller | - ' ' 2026-03-20 16:26:25.819173 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.819179 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.819184 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.819190 | controller | - ' ' 2026-03-20 16:26:25.819196 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.819202 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.819212 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.819217 | controller | available"' 2026-03-20 16:26:25.819223 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.819240 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.819248 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.819254 | controller | - ' exit 1' 2026-03-20 16:26:25.819260 | controller | - ' fi' 2026-03-20 16:26:25.819266 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.819272 | controller | - ' ' 2026-03-20 16:26:25.819277 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.819283 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.819289 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.819295 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.819301 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.819307 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.819312 | controller | -- \' 2026-03-20 16:26:25.819318 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.819324 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.819340 | controller | - ' logger.go:42: 16:16:37 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.819346 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.819355 | controller | - ' logger.go:42: 16:16:37 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.819361 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.819367 | controller | found' 2026-03-20 16:26:25.819373 | controller | - ' logger.go:42: 16:16:38 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.819379 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.819385 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.819391 | controller | - ' ' 2026-03-20 16:26:25.819397 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.819403 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.819409 | controller | - ' ' 2026-03-20 16:26:25.819414 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.819420 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.819426 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.819432 | controller | - ' exit 1' 2026-03-20 16:26:25.819437 | controller | - ' fi' 2026-03-20 16:26:25.819443 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.819449 | controller | - ' ' 2026-03-20 16:26:25.819455 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.819461 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.819466 | controller | | base64 -d)' 2026-03-20 16:26:25.819472 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.819478 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.819484 | controller | - ' exit 1' 2026-03-20 16:26:25.819490 | controller | - ' fi' 2026-03-20 16:26:25.819495 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.819501 | controller | - ' ' 2026-03-20 16:26:25.819507 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.819513 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.819518 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.819527 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.819533 | controller | --timeout=300s' 2026-03-20 16:26:25.819538 | controller | - ' ' 2026-03-20 16:26:25.819547 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.819553 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.819559 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.819565 | controller | - ' ' 2026-03-20 16:26:25.819571 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.819579 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.819585 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.819590 | controller | - ' ' 2026-03-20 16:26:25.819596 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.819602 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.819610 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.819615 | controller | available"' 2026-03-20 16:26:25.819621 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.819627 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.819633 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.819639 | controller | - ' exit 1' 2026-03-20 16:26:25.819645 | controller | - ' fi' 2026-03-20 16:26:25.819651 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.819656 | controller | - ' ' 2026-03-20 16:26:25.819662 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.819668 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.819674 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.819680 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.819685 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.819711 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.819721 | controller | -- \' 2026-03-20 16:26:25.819727 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.819752 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.819758 | controller | - ' logger.go:42: 16:16:38 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.819774 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.819780 | controller | - ' logger.go:42: 16:16:39 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.819786 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.819792 | controller | found' 2026-03-20 16:26:25.819798 | controller | - ' logger.go:42: 16:16:40 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.819803 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.819809 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.819815 | controller | - ' ' 2026-03-20 16:26:25.819821 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.819830 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.819836 | controller | - ' ' 2026-03-20 16:26:25.819842 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.819847 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.819853 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.819859 | controller | - ' exit 1' 2026-03-20 16:26:25.819865 | controller | - ' fi' 2026-03-20 16:26:25.819871 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.819877 | controller | - ' ' 2026-03-20 16:26:25.819883 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.819888 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.819894 | controller | | base64 -d)' 2026-03-20 16:26:25.819900 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.819906 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.819911 | controller | - ' exit 1' 2026-03-20 16:26:25.819917 | controller | - ' fi' 2026-03-20 16:26:25.819923 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.819929 | controller | - ' ' 2026-03-20 16:26:25.819935 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.819940 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.819946 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.819952 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.819958 | controller | --timeout=300s' 2026-03-20 16:26:25.819964 | controller | - ' ' 2026-03-20 16:26:25.819969 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.819975 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.819981 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.819986 | controller | - ' ' 2026-03-20 16:26:25.819992 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.819998 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.820004 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.820009 | controller | - ' ' 2026-03-20 16:26:25.820015 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.820021 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.820026 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.820032 | controller | available"' 2026-03-20 16:26:25.820038 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.820047 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.820053 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.820059 | controller | - ' exit 1' 2026-03-20 16:26:25.820065 | controller | - ' fi' 2026-03-20 16:26:25.820071 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.820076 | controller | - ' ' 2026-03-20 16:26:25.820082 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.820088 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.820096 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.820102 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.820108 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.820114 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.820120 | controller | -- \' 2026-03-20 16:26:25.820125 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.820131 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.820137 | controller | - ' logger.go:42: 16:16:40 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.820143 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.820160 | controller | - ' logger.go:42: 16:16:40 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.820166 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.820171 | controller | found' 2026-03-20 16:26:25.820177 | controller | - ' logger.go:42: 16:16:41 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.820183 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.820189 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.820194 | controller | - ' ' 2026-03-20 16:26:25.820200 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.820206 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.820212 | controller | - ' ' 2026-03-20 16:26:25.820218 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.820223 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.820229 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.820235 | controller | - ' exit 1' 2026-03-20 16:26:25.820241 | controller | - ' fi' 2026-03-20 16:26:25.820246 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.820252 | controller | - ' ' 2026-03-20 16:26:25.820258 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.820263 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.820269 | controller | | base64 -d)' 2026-03-20 16:26:25.820275 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.820281 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.820287 | controller | - ' exit 1' 2026-03-20 16:26:25.820292 | controller | - ' fi' 2026-03-20 16:26:25.820298 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.820305 | controller | - ' ' 2026-03-20 16:26:25.820312 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.820318 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.820325 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.820332 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.820338 | controller | --timeout=300s' 2026-03-20 16:26:25.820344 | controller | - ' ' 2026-03-20 16:26:25.820349 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.820355 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.820361 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.820370 | controller | - ' ' 2026-03-20 16:26:25.820376 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.820382 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.820388 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.820393 | controller | - ' ' 2026-03-20 16:26:25.820399 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.820405 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.820411 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.820417 | controller | available"' 2026-03-20 16:26:25.820423 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.820428 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.820434 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.820440 | controller | - ' exit 1' 2026-03-20 16:26:25.820446 | controller | - ' fi' 2026-03-20 16:26:25.820452 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.820457 | controller | - ' ' 2026-03-20 16:26:25.820463 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.820469 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.820475 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.820481 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.820487 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.820492 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.820498 | controller | -- \' 2026-03-20 16:26:25.820504 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.820510 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.820515 | controller | - ' logger.go:42: 16:16:41 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.820521 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.820527 | controller | - ' logger.go:42: 16:16:41 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.820544 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.820551 | controller | found' 2026-03-20 16:26:25.820556 | controller | - ' logger.go:42: 16:16:42 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.820562 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.820568 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.820574 | controller | - ' ' 2026-03-20 16:26:25.820580 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.820586 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.820591 | controller | - ' ' 2026-03-20 16:26:25.820597 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.820603 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.820609 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.820614 | controller | - ' exit 1' 2026-03-20 16:26:25.820620 | controller | - ' fi' 2026-03-20 16:26:25.820626 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.820635 | controller | - ' ' 2026-03-20 16:26:25.820641 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.820646 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.820652 | controller | | base64 -d)' 2026-03-20 16:26:25.820658 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.820664 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.820670 | controller | - ' exit 1' 2026-03-20 16:26:25.820675 | controller | - ' fi' 2026-03-20 16:26:25.820681 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.820687 | controller | - ' ' 2026-03-20 16:26:25.820708 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.820718 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.820724 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.820730 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.820735 | controller | --timeout=300s' 2026-03-20 16:26:25.820741 | controller | - ' ' 2026-03-20 16:26:25.820747 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.820752 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.820758 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.820763 | controller | - ' ' 2026-03-20 16:26:25.820769 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.820775 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.820780 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.820786 | controller | - ' ' 2026-03-20 16:26:25.820791 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.820797 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.820802 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.820808 | controller | available"' 2026-03-20 16:26:25.820814 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.820824 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.820830 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.820836 | controller | - ' exit 1' 2026-03-20 16:26:25.820841 | controller | - ' fi' 2026-03-20 16:26:25.820847 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.820853 | controller | - ' ' 2026-03-20 16:26:25.820861 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.820867 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.820873 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.820878 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.820884 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.820890 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.820901 | controller | -- \' 2026-03-20 16:26:25.820907 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.820917 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.820922 | controller | - ' logger.go:42: 16:16:42 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.820928 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.820934 | controller | - ' logger.go:42: 16:16:42 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.820951 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.820958 | controller | found' 2026-03-20 16:26:25.820964 | controller | - ' logger.go:42: 16:16:43 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.820970 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.820976 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.820982 | controller | - ' ' 2026-03-20 16:26:25.820987 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.820993 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.820999 | controller | - ' ' 2026-03-20 16:26:25.821005 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.821010 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.821016 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.821022 | controller | - ' exit 1' 2026-03-20 16:26:25.821028 | controller | - ' fi' 2026-03-20 16:26:25.821034 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.821039 | controller | - ' ' 2026-03-20 16:26:25.821045 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.821051 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.821056 | controller | | base64 -d)' 2026-03-20 16:26:25.821063 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.821067 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.821071 | controller | - ' exit 1' 2026-03-20 16:26:25.821075 | controller | - ' fi' 2026-03-20 16:26:25.821080 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.821084 | controller | - ' ' 2026-03-20 16:26:25.821088 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.821092 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.821096 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.821100 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.821105 | controller | --timeout=300s' 2026-03-20 16:26:25.821109 | controller | - ' ' 2026-03-20 16:26:25.821113 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.821117 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.821121 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.821125 | controller | - ' ' 2026-03-20 16:26:25.821130 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.821134 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.821138 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.821142 | controller | - ' ' 2026-03-20 16:26:25.821146 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.821153 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.821157 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.821161 | controller | available"' 2026-03-20 16:26:25.821165 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.821169 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.821174 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.821178 | controller | - ' exit 1' 2026-03-20 16:26:25.821182 | controller | - ' fi' 2026-03-20 16:26:25.821186 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.821190 | controller | - ' ' 2026-03-20 16:26:25.821195 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.821199 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.821203 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.821207 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.821211 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.821215 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.821219 | controller | -- \' 2026-03-20 16:26:25.821224 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.821228 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.821232 | controller | - ' logger.go:42: 16:16:43 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.821236 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.821240 | controller | - ' logger.go:42: 16:16:43 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.821245 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.821249 | controller | found' 2026-03-20 16:26:25.821261 | controller | - ' logger.go:42: 16:16:44 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.821266 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.821270 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.821274 | controller | - ' ' 2026-03-20 16:26:25.821279 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.821283 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.821287 | controller | - ' ' 2026-03-20 16:26:25.821291 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.821295 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.821300 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.821304 | controller | - ' exit 1' 2026-03-20 16:26:25.821308 | controller | - ' fi' 2026-03-20 16:26:25.821312 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.821316 | controller | - ' ' 2026-03-20 16:26:25.821320 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.821325 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.821329 | controller | | base64 -d)' 2026-03-20 16:26:25.821333 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.821337 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.821341 | controller | - ' exit 1' 2026-03-20 16:26:25.821349 | controller | - ' fi' 2026-03-20 16:26:25.821353 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.821359 | controller | - ' ' 2026-03-20 16:26:25.821363 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.821367 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.821371 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.821376 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.821380 | controller | --timeout=300s' 2026-03-20 16:26:25.821384 | controller | - ' ' 2026-03-20 16:26:25.821388 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.821392 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.821397 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.821401 | controller | - ' ' 2026-03-20 16:26:25.821405 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.821410 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.821416 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.821421 | controller | - ' ' 2026-03-20 16:26:25.821427 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.821433 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.821438 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.821444 | controller | available"' 2026-03-20 16:26:25.821450 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.821455 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.821461 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.821467 | controller | - ' exit 1' 2026-03-20 16:26:25.821472 | controller | - ' fi' 2026-03-20 16:26:25.821478 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.821483 | controller | - ' ' 2026-03-20 16:26:25.821489 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.821494 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.821499 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.821508 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.821514 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.821519 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.821523 | controller | -- \' 2026-03-20 16:26:25.821527 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.821531 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.821536 | controller | - ' logger.go:42: 16:16:44 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.821540 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.821544 | controller | - ' logger.go:42: 16:16:44 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.821548 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.821554 | controller | found' 2026-03-20 16:26:25.821568 | controller | - ' logger.go:42: 16:16:45 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.821574 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.821578 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.821582 | controller | - ' ' 2026-03-20 16:26:25.821586 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.821591 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.821595 | controller | - ' ' 2026-03-20 16:26:25.821599 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.821603 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.821607 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.821611 | controller | - ' exit 1' 2026-03-20 16:26:25.821616 | controller | - ' fi' 2026-03-20 16:26:25.821620 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.821624 | controller | - ' ' 2026-03-20 16:26:25.821628 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.821632 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.821637 | controller | | base64 -d)' 2026-03-20 16:26:25.821641 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.821645 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.821649 | controller | - ' exit 1' 2026-03-20 16:26:25.821653 | controller | - ' fi' 2026-03-20 16:26:25.821657 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.821662 | controller | - ' ' 2026-03-20 16:26:25.821666 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.821670 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.821674 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.821678 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.821683 | controller | --timeout=300s' 2026-03-20 16:26:25.821689 | controller | - ' ' 2026-03-20 16:26:25.821706 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.821715 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.821719 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.821723 | controller | - ' ' 2026-03-20 16:26:25.821727 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.821731 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.821736 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.821740 | controller | - ' ' 2026-03-20 16:26:25.821744 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.821748 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.821752 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.821756 | controller | available"' 2026-03-20 16:26:25.821761 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.821765 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.821769 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.821775 | controller | - ' exit 1' 2026-03-20 16:26:25.821780 | controller | - ' fi' 2026-03-20 16:26:25.821784 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.821788 | controller | - ' ' 2026-03-20 16:26:25.821792 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.821797 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.821801 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.821805 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.821809 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.821813 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.821817 | controller | -- \' 2026-03-20 16:26:25.821822 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.821826 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.821830 | controller | - ' logger.go:42: 16:16:45 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.821834 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.821838 | controller | - ' logger.go:42: 16:16:46 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.821843 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.821847 | controller | found' 2026-03-20 16:26:25.821851 | controller | - ' logger.go:42: 16:16:47 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.821855 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.821868 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.821872 | controller | - ' ' 2026-03-20 16:26:25.821877 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.821881 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.821885 | controller | - ' ' 2026-03-20 16:26:25.821890 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.821894 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.821898 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.821902 | controller | - ' exit 1' 2026-03-20 16:26:25.821907 | controller | - ' fi' 2026-03-20 16:26:25.821911 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.821915 | controller | - ' ' 2026-03-20 16:26:25.821919 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.821923 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.821930 | controller | | base64 -d)' 2026-03-20 16:26:25.821934 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.821938 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.821943 | controller | - ' exit 1' 2026-03-20 16:26:25.821947 | controller | - ' fi' 2026-03-20 16:26:25.821951 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.821955 | controller | - ' ' 2026-03-20 16:26:25.821959 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.821965 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.821970 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.821976 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.821984 | controller | --timeout=300s' 2026-03-20 16:26:25.821990 | controller | - ' ' 2026-03-20 16:26:25.821996 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.822001 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.822007 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.822013 | controller | - ' ' 2026-03-20 16:26:25.822019 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.822024 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.822033 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.822039 | controller | - ' ' 2026-03-20 16:26:25.822045 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.822051 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.822057 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.822063 | controller | available"' 2026-03-20 16:26:25.822068 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.822074 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.822080 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.822085 | controller | - ' exit 1' 2026-03-20 16:26:25.822091 | controller | - ' fi' 2026-03-20 16:26:25.822097 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.822103 | controller | - ' ' 2026-03-20 16:26:25.822108 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.822114 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.822119 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.822125 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.822131 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.822136 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.822142 | controller | -- \' 2026-03-20 16:26:25.822147 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.822156 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.822162 | controller | - ' logger.go:42: 16:16:47 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.822168 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.822174 | controller | - ' logger.go:42: 16:16:47 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.822179 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.822185 | controller | found' 2026-03-20 16:26:25.822191 | controller | - ' logger.go:42: 16:16:48 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.822196 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.822201 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.822207 | controller | - ' ' 2026-03-20 16:26:25.822227 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.822234 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.822244 | controller | - ' ' 2026-03-20 16:26:25.822250 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.822255 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.822261 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.822267 | controller | - ' exit 1' 2026-03-20 16:26:25.822273 | controller | - ' fi' 2026-03-20 16:26:25.822279 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.822285 | controller | - ' ' 2026-03-20 16:26:25.822294 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.822300 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.822306 | controller | | base64 -d)' 2026-03-20 16:26:25.822312 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.822318 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.822324 | controller | - ' exit 1' 2026-03-20 16:26:25.822330 | controller | - ' fi' 2026-03-20 16:26:25.822336 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.822341 | controller | - ' ' 2026-03-20 16:26:25.822347 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.822353 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.822359 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.822365 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.822371 | controller | --timeout=300s' 2026-03-20 16:26:25.822377 | controller | - ' ' 2026-03-20 16:26:25.822383 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.822388 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.822394 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.822400 | controller | - ' ' 2026-03-20 16:26:25.822406 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.822412 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.822418 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.822423 | controller | - ' ' 2026-03-20 16:26:25.822429 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.822435 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.822440 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.822446 | controller | available"' 2026-03-20 16:26:25.822452 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.822458 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.822463 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.822469 | controller | - ' exit 1' 2026-03-20 16:26:25.822475 | controller | - ' fi' 2026-03-20 16:26:25.822481 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.822486 | controller | - ' ' 2026-03-20 16:26:25.822492 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.822498 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.822504 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.822514 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.822520 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.822526 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.822532 | controller | -- \' 2026-03-20 16:26:25.822539 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.822544 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.822550 | controller | - ' logger.go:42: 16:16:48 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.822556 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.822562 | controller | - ' logger.go:42: 16:16:48 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.822568 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.822574 | controller | found' 2026-03-20 16:26:25.822579 | controller | - ' logger.go:42: 16:16:49 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.822584 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.822590 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.822595 | controller | - ' ' 2026-03-20 16:26:25.822613 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.822620 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.822625 | controller | - ' ' 2026-03-20 16:26:25.822631 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.822637 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.822642 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.822648 | controller | - ' exit 1' 2026-03-20 16:26:25.822653 | controller | - ' fi' 2026-03-20 16:26:25.822659 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.822665 | controller | - ' ' 2026-03-20 16:26:25.822670 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.822676 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.822682 | controller | | base64 -d)' 2026-03-20 16:26:25.822687 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.822709 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.822719 | controller | - ' exit 1' 2026-03-20 16:26:25.822725 | controller | - ' fi' 2026-03-20 16:26:25.822731 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.822738 | controller | - ' ' 2026-03-20 16:26:25.822744 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.822750 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.822756 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.822762 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.822768 | controller | --timeout=300s' 2026-03-20 16:26:25.822774 | controller | - ' ' 2026-03-20 16:26:25.822780 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.822786 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.822792 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.822799 | controller | - ' ' 2026-03-20 16:26:25.822809 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.822815 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.822820 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.822826 | controller | - ' ' 2026-03-20 16:26:25.822831 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.822837 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.822843 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.822848 | controller | available"' 2026-03-20 16:26:25.822854 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.822860 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.822865 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.822870 | controller | - ' exit 1' 2026-03-20 16:26:25.822876 | controller | - ' fi' 2026-03-20 16:26:25.822882 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.822887 | controller | - ' ' 2026-03-20 16:26:25.822892 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.822898 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.822903 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.822909 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.822915 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.822920 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.822926 | controller | -- \' 2026-03-20 16:26:25.822931 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.822937 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.822943 | controller | - ' logger.go:42: 16:16:49 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.822948 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.822957 | controller | - ' logger.go:42: 16:16:49 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.822963 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.822969 | controller | found' 2026-03-20 16:26:25.822975 | controller | - ' logger.go:42: 16:16:50 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.822981 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.822987 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.822992 | controller | - ' ' 2026-03-20 16:26:25.822998 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.823017 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.823023 | controller | - ' ' 2026-03-20 16:26:25.823030 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.823036 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.823041 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.823051 | controller | - ' exit 1' 2026-03-20 16:26:25.823057 | controller | - ' fi' 2026-03-20 16:26:25.823063 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.823069 | controller | - ' ' 2026-03-20 16:26:25.823078 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.823084 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.823090 | controller | | base64 -d)' 2026-03-20 16:26:25.823096 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.823102 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.823108 | controller | - ' exit 1' 2026-03-20 16:26:25.823114 | controller | - ' fi' 2026-03-20 16:26:25.823120 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.823126 | controller | - ' ' 2026-03-20 16:26:25.823132 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.823138 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.823144 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.823150 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.823156 | controller | --timeout=300s' 2026-03-20 16:26:25.823163 | controller | - ' ' 2026-03-20 16:26:25.823169 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.823175 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.823181 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.823186 | controller | - ' ' 2026-03-20 16:26:25.823192 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.823198 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.823203 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.823209 | controller | - ' ' 2026-03-20 16:26:25.823215 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.823221 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.823226 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.823232 | controller | available"' 2026-03-20 16:26:25.823238 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.823244 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.823250 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.823256 | controller | - ' exit 1' 2026-03-20 16:26:25.823262 | controller | - ' fi' 2026-03-20 16:26:25.823268 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.823273 | controller | - ' ' 2026-03-20 16:26:25.823279 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.823284 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.823290 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.823296 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.823302 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.823308 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.823314 | controller | -- \' 2026-03-20 16:26:25.823320 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.823329 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.823338 | controller | - ' logger.go:42: 16:16:50 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.823344 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.823351 | controller | - ' logger.go:42: 16:16:50 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.823356 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.823362 | controller | found' 2026-03-20 16:26:25.823368 | controller | - ' logger.go:42: 16:16:51 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.823373 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.823379 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.823385 | controller | - ' ' 2026-03-20 16:26:25.823390 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.823409 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.823416 | controller | - ' ' 2026-03-20 16:26:25.823422 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.823428 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.823434 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.823439 | controller | - ' exit 1' 2026-03-20 16:26:25.823445 | controller | - ' fi' 2026-03-20 16:26:25.823451 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.823457 | controller | - ' ' 2026-03-20 16:26:25.823463 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.823469 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.823475 | controller | | base64 -d)' 2026-03-20 16:26:25.823481 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.823487 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.823493 | controller | - ' exit 1' 2026-03-20 16:26:25.823499 | controller | - ' fi' 2026-03-20 16:26:25.823505 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.823511 | controller | - ' ' 2026-03-20 16:26:25.823516 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.823522 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.823527 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.823534 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.823540 | controller | --timeout=300s' 2026-03-20 16:26:25.823546 | controller | - ' ' 2026-03-20 16:26:25.823552 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.823558 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.823563 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.823569 | controller | - ' ' 2026-03-20 16:26:25.823575 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.823581 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.823587 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.823592 | controller | - ' ' 2026-03-20 16:26:25.823602 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.823611 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.823617 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.823623 | controller | available"' 2026-03-20 16:26:25.823629 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.823635 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.823641 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.823647 | controller | - ' exit 1' 2026-03-20 16:26:25.823653 | controller | - ' fi' 2026-03-20 16:26:25.823659 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.823665 | controller | - ' ' 2026-03-20 16:26:25.823671 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.823677 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.823683 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.823689 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.823717 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.823724 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.823730 | controller | -- \' 2026-03-20 16:26:25.823757 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.823764 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.823770 | controller | - ' logger.go:42: 16:16:51 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.823776 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.823782 | controller | - ' logger.go:42: 16:16:51 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.823793 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.823800 | controller | found' 2026-03-20 16:26:25.823805 | controller | - ' logger.go:42: 16:16:52 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.823811 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.823816 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.823825 | controller | - ' ' 2026-03-20 16:26:25.823832 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.823837 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.823843 | controller | - ' ' 2026-03-20 16:26:25.823862 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.823870 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.823876 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.823882 | controller | - ' exit 1' 2026-03-20 16:26:25.823887 | controller | - ' fi' 2026-03-20 16:26:25.823893 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.823899 | controller | - ' ' 2026-03-20 16:26:25.823905 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.823911 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.823917 | controller | | base64 -d)' 2026-03-20 16:26:25.823923 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.823928 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.823934 | controller | - ' exit 1' 2026-03-20 16:26:25.823940 | controller | - ' fi' 2026-03-20 16:26:25.823950 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.823956 | controller | - ' ' 2026-03-20 16:26:25.823962 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.823968 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.823974 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.823980 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.823986 | controller | --timeout=300s' 2026-03-20 16:26:25.823993 | controller | - ' ' 2026-03-20 16:26:25.823998 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.824004 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.824011 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.824017 | controller | - ' ' 2026-03-20 16:26:25.824023 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.824029 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.824034 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.824040 | controller | - ' ' 2026-03-20 16:26:25.824046 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.824052 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.824058 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.824064 | controller | available"' 2026-03-20 16:26:25.824070 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.824080 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.824086 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.824092 | controller | - ' exit 1' 2026-03-20 16:26:25.824098 | controller | - ' fi' 2026-03-20 16:26:25.824105 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.824111 | controller | - ' ' 2026-03-20 16:26:25.824117 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.824124 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.824130 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.824136 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.824142 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.824149 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.824155 | controller | -- \' 2026-03-20 16:26:25.824264 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.824274 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.824281 | controller | - ' logger.go:42: 16:16:52 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.824287 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.824293 | controller | - ' logger.go:42: 16:16:53 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.824300 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.824312 | controller | found' 2026-03-20 16:26:25.824318 | controller | - ' logger.go:42: 16:16:54 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.824324 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.824330 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.824336 | controller | - ' ' 2026-03-20 16:26:25.824341 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.824347 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.824352 | controller | - ' ' 2026-03-20 16:26:25.824371 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.824377 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.824384 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.824390 | controller | - ' exit 1' 2026-03-20 16:26:25.824396 | controller | - ' fi' 2026-03-20 16:26:25.824402 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.824408 | controller | - ' ' 2026-03-20 16:26:25.824415 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.824420 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.824426 | controller | | base64 -d)' 2026-03-20 16:26:25.824432 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.824438 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.824445 | controller | - ' exit 1' 2026-03-20 16:26:25.824451 | controller | - ' fi' 2026-03-20 16:26:25.824457 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.824463 | controller | - ' ' 2026-03-20 16:26:25.824469 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.824475 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.824480 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.824487 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.824493 | controller | --timeout=300s' 2026-03-20 16:26:25.824498 | controller | - ' ' 2026-03-20 16:26:25.824504 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.824510 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.824516 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.824522 | controller | - ' ' 2026-03-20 16:26:25.824529 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.824535 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.824541 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.824547 | controller | - ' ' 2026-03-20 16:26:25.824553 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.824559 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.824565 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.824571 | controller | available"' 2026-03-20 16:26:25.824577 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.824583 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.824589 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.824595 | controller | - ' exit 1' 2026-03-20 16:26:25.824605 | controller | - ' fi' 2026-03-20 16:26:25.824612 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.824618 | controller | - ' ' 2026-03-20 16:26:25.824624 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.824630 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.824637 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.824643 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.824648 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.824654 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.824659 | controller | -- \' 2026-03-20 16:26:25.824664 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.824670 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.824676 | controller | - ' logger.go:42: 16:16:54 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.824681 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.824687 | controller | - ' logger.go:42: 16:16:54 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.824710 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.824721 | controller | found' 2026-03-20 16:26:25.824731 | controller | - ' logger.go:42: 16:16:55 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.824737 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.824743 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.824748 | controller | - ' ' 2026-03-20 16:26:25.824754 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.824759 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.824765 | controller | - ' ' 2026-03-20 16:26:25.824771 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.824788 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.824795 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.824801 | controller | - ' exit 1' 2026-03-20 16:26:25.824807 | controller | - ' fi' 2026-03-20 16:26:25.824813 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.824819 | controller | - ' ' 2026-03-20 16:26:25.824825 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.824831 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.824837 | controller | | base64 -d)' 2026-03-20 16:26:25.824842 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.824848 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.824854 | controller | - ' exit 1' 2026-03-20 16:26:25.824858 | controller | - ' fi' 2026-03-20 16:26:25.824862 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.824866 | controller | - ' ' 2026-03-20 16:26:25.824871 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.824875 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.824879 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.824883 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.824889 | controller | --timeout=300s' 2026-03-20 16:26:25.824899 | controller | - ' ' 2026-03-20 16:26:25.824905 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.824911 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.824917 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.824922 | controller | - ' ' 2026-03-20 16:26:25.824928 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.824934 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.824939 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.824945 | controller | - ' ' 2026-03-20 16:26:25.824951 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.824973 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.824979 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.824985 | controller | available"' 2026-03-20 16:26:25.824991 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.824997 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.825002 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.825008 | controller | - ' exit 1' 2026-03-20 16:26:25.825014 | controller | - ' fi' 2026-03-20 16:26:25.825020 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.825025 | controller | - ' ' 2026-03-20 16:26:25.825031 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.825036 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.825042 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.825048 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.825054 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.825063 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.825068 | controller | -- \' 2026-03-20 16:26:25.825077 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.825082 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.825088 | controller | - ' logger.go:42: 16:16:55 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.825094 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.825100 | controller | - ' logger.go:42: 16:16:55 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.825106 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.825112 | controller | found' 2026-03-20 16:26:25.825118 | controller | - ' logger.go:42: 16:16:56 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.825124 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.825130 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.825136 | controller | - ' ' 2026-03-20 16:26:25.825141 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.825147 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.825153 | controller | - ' ' 2026-03-20 16:26:25.825163 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.825169 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.825187 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.825194 | controller | - ' exit 1' 2026-03-20 16:26:25.825200 | controller | - ' fi' 2026-03-20 16:26:25.825207 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.825213 | controller | - ' ' 2026-03-20 16:26:25.825219 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.825225 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.825231 | controller | | base64 -d)' 2026-03-20 16:26:25.825237 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.825243 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.825249 | controller | - ' exit 1' 2026-03-20 16:26:25.825255 | controller | - ' fi' 2026-03-20 16:26:25.825265 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.825271 | controller | - ' ' 2026-03-20 16:26:25.825276 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.825282 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.825289 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.825295 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.825301 | controller | --timeout=300s' 2026-03-20 16:26:25.825307 | controller | - ' ' 2026-03-20 16:26:25.825313 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.825320 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.825326 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.825335 | controller | - ' ' 2026-03-20 16:26:25.825342 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.825348 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.825354 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.825361 | controller | - ' ' 2026-03-20 16:26:25.825367 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.825373 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.825379 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.825386 | controller | available"' 2026-03-20 16:26:25.825392 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.825398 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.825404 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.825410 | controller | - ' exit 1' 2026-03-20 16:26:25.825415 | controller | - ' fi' 2026-03-20 16:26:25.825421 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.825427 | controller | - ' ' 2026-03-20 16:26:25.825433 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.825439 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.825445 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.825454 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.825460 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.825466 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.825472 | controller | -- \' 2026-03-20 16:26:25.825478 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.825485 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.825490 | controller | - ' logger.go:42: 16:16:56 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.825496 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.825502 | controller | - ' logger.go:42: 16:16:56 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.825509 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.825514 | controller | found' 2026-03-20 16:26:25.825520 | controller | - ' logger.go:42: 16:16:57 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.825527 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.825533 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.825539 | controller | - ' ' 2026-03-20 16:26:25.825549 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.825556 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.825562 | controller | - ' ' 2026-03-20 16:26:25.825568 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.825574 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.825580 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.825600 | controller | - ' exit 1' 2026-03-20 16:26:25.825608 | controller | - ' fi' 2026-03-20 16:26:25.825615 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.825620 | controller | - ' ' 2026-03-20 16:26:25.825626 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.825632 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.825639 | controller | | base64 -d)' 2026-03-20 16:26:25.825645 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.825652 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.825658 | controller | - ' exit 1' 2026-03-20 16:26:25.825664 | controller | - ' fi' 2026-03-20 16:26:25.825669 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.825675 | controller | - ' ' 2026-03-20 16:26:25.825681 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.825688 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.825718 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.825730 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.825736 | controller | --timeout=300s' 2026-03-20 16:26:25.825742 | controller | - ' ' 2026-03-20 16:26:25.825748 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.825754 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.825760 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.825766 | controller | - ' ' 2026-03-20 16:26:25.825772 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.825793 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.825800 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.825806 | controller | - ' ' 2026-03-20 16:26:25.825812 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.825818 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.825824 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.825830 | controller | available"' 2026-03-20 16:26:25.825836 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.825842 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.825848 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.825854 | controller | - ' exit 1' 2026-03-20 16:26:25.825860 | controller | - ' fi' 2026-03-20 16:26:25.825865 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.825871 | controller | - ' ' 2026-03-20 16:26:25.825877 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.825883 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.825889 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.825895 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.825901 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.825907 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.825913 | controller | -- \' 2026-03-20 16:26:25.825919 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.825925 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.825931 | controller | - ' logger.go:42: 16:16:57 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.825937 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.825943 | controller | - ' logger.go:42: 16:16:57 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.825949 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.825955 | controller | found' 2026-03-20 16:26:25.825960 | controller | - ' logger.go:42: 16:16:58 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.825966 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.825972 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.825978 | controller | - ' ' 2026-03-20 16:26:25.825984 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.825989 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.825994 | controller | - ' ' 2026-03-20 16:26:25.825998 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.826002 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.826007 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.826012 | controller | - ' exit 1' 2026-03-20 16:26:25.826018 | controller | - ' fi' 2026-03-20 16:26:25.826036 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.826043 | controller | - ' ' 2026-03-20 16:26:25.826053 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.826071 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.826077 | controller | | base64 -d)' 2026-03-20 16:26:25.826083 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.826087 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.826091 | controller | - ' exit 1' 2026-03-20 16:26:25.826095 | controller | - ' fi' 2026-03-20 16:26:25.826101 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.826107 | controller | - ' ' 2026-03-20 16:26:25.826112 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.826118 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.826124 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.826129 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.826135 | controller | --timeout=300s' 2026-03-20 16:26:25.826141 | controller | - ' ' 2026-03-20 16:26:25.826147 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.826152 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.826158 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.826164 | controller | - ' ' 2026-03-20 16:26:25.826169 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.826175 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.826181 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.826187 | controller | - ' ' 2026-03-20 16:26:25.826192 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.826198 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.826204 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.826210 | controller | available"' 2026-03-20 16:26:25.826215 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.826221 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.826227 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.826232 | controller | - ' exit 1' 2026-03-20 16:26:25.826238 | controller | - ' fi' 2026-03-20 16:26:25.826244 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.826250 | controller | - ' ' 2026-03-20 16:26:25.826255 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.826261 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.826266 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.826270 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.826274 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.826279 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.826283 | controller | -- \' 2026-03-20 16:26:25.826287 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.826291 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.826298 | controller | - ' logger.go:42: 16:16:58 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.826302 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.826306 | controller | - ' logger.go:42: 16:16:58 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.826311 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.826315 | controller | found' 2026-03-20 16:26:25.826319 | controller | - ' logger.go:42: 16:16:59 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.826323 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.826329 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.826334 | controller | - ' ' 2026-03-20 16:26:25.826339 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.826349 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.826354 | controller | - ' ' 2026-03-20 16:26:25.826358 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.826363 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.826367 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.826371 | controller | - ' exit 1' 2026-03-20 16:26:25.826375 | controller | - ' fi' 2026-03-20 16:26:25.826379 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.826384 | controller | - ' ' 2026-03-20 16:26:25.826397 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.826401 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.826405 | controller | | base64 -d)' 2026-03-20 16:26:25.826410 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.826414 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.826418 | controller | - ' exit 1' 2026-03-20 16:26:25.826422 | controller | - ' fi' 2026-03-20 16:26:25.826426 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.826431 | controller | - ' ' 2026-03-20 16:26:25.826435 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.826439 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.826443 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.826447 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.826451 | controller | --timeout=300s' 2026-03-20 16:26:25.826456 | controller | - ' ' 2026-03-20 16:26:25.826460 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.826464 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.826470 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.826474 | controller | - ' ' 2026-03-20 16:26:25.826479 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.826483 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.826488 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.826494 | controller | - ' ' 2026-03-20 16:26:25.826500 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.826505 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.826513 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.826518 | controller | available"' 2026-03-20 16:26:25.826522 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.826526 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.826530 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.826534 | controller | - ' exit 1' 2026-03-20 16:26:25.826538 | controller | - ' fi' 2026-03-20 16:26:25.826543 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.826547 | controller | - ' ' 2026-03-20 16:26:25.826551 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.826555 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.826559 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.826563 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.826568 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.826572 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.826576 | controller | -- \' 2026-03-20 16:26:25.826580 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.826584 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.826589 | controller | - ' logger.go:42: 16:16:59 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.826593 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.826597 | controller | - ' logger.go:42: 16:17:00 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.826601 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.826605 | controller | found' 2026-03-20 16:26:25.826609 | controller | - ' logger.go:42: 16:17:01 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.826613 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.826618 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.826622 | controller | - ' ' 2026-03-20 16:26:25.826626 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.826630 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.826634 | controller | - ' ' 2026-03-20 16:26:25.826638 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.826643 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.826647 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.826651 | controller | - ' exit 1' 2026-03-20 16:26:25.826658 | controller | - ' fi' 2026-03-20 16:26:25.826662 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.826668 | controller | - ' ' 2026-03-20 16:26:25.826683 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.826689 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.826715 | controller | | base64 -d)' 2026-03-20 16:26:25.826720 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.826725 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.826729 | controller | - ' exit 1' 2026-03-20 16:26:25.826733 | controller | - ' fi' 2026-03-20 16:26:25.826737 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.826750 | controller | - ' ' 2026-03-20 16:26:25.826756 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.826762 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.826768 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.826773 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.826779 | controller | --timeout=300s' 2026-03-20 16:26:25.826785 | controller | - ' ' 2026-03-20 16:26:25.826791 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.826796 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.826802 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.826808 | controller | - ' ' 2026-03-20 16:26:25.826814 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.826820 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.826825 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.826831 | controller | - ' ' 2026-03-20 16:26:25.826837 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.826842 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.826848 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.826854 | controller | available"' 2026-03-20 16:26:25.826860 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.826866 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.826872 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.826877 | controller | - ' exit 1' 2026-03-20 16:26:25.826883 | controller | - ' fi' 2026-03-20 16:26:25.826889 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.826894 | controller | - ' ' 2026-03-20 16:26:25.826900 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.826906 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.826912 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.826918 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.826923 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.826929 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.826935 | controller | -- \' 2026-03-20 16:26:25.826940 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.826946 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.826952 | controller | - ' logger.go:42: 16:17:01 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.826958 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.826964 | controller | - ' logger.go:42: 16:17:01 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.826969 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.826975 | controller | found' 2026-03-20 16:26:25.826981 | controller | - ' logger.go:42: 16:17:02 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.826991 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.826997 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.827003 | controller | - ' ' 2026-03-20 16:26:25.827008 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.827014 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.827020 | controller | - ' ' 2026-03-20 16:26:25.827039 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.827046 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.827052 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.827058 | controller | - ' exit 1' 2026-03-20 16:26:25.827063 | controller | - ' fi' 2026-03-20 16:26:25.827069 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.827075 | controller | - ' ' 2026-03-20 16:26:25.827081 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.827100 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.827106 | controller | | base64 -d)' 2026-03-20 16:26:25.827112 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.827118 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.827124 | controller | - ' exit 1' 2026-03-20 16:26:25.827129 | controller | - ' fi' 2026-03-20 16:26:25.827135 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.827141 | controller | - ' ' 2026-03-20 16:26:25.827147 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.827152 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.827158 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.827164 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.827169 | controller | --timeout=300s' 2026-03-20 16:26:25.827179 | controller | - ' ' 2026-03-20 16:26:25.827184 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.827190 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.827196 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.827202 | controller | - ' ' 2026-03-20 16:26:25.827208 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.827213 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.827219 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.827225 | controller | - ' ' 2026-03-20 16:26:25.827231 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.827236 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.827242 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.827248 | controller | available"' 2026-03-20 16:26:25.827254 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.827260 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.827266 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.827271 | controller | - ' exit 1' 2026-03-20 16:26:25.827278 | controller | - ' fi' 2026-03-20 16:26:25.827287 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.827293 | controller | - ' ' 2026-03-20 16:26:25.827300 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.827306 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.827312 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.827317 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.827323 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.827329 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.827334 | controller | -- \' 2026-03-20 16:26:25.827340 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.827345 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.827351 | controller | - ' logger.go:42: 16:17:02 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.827357 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.827363 | controller | - ' logger.go:42: 16:17:02 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.827368 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.827374 | controller | found' 2026-03-20 16:26:25.827380 | controller | - ' logger.go:42: 16:17:03 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.827386 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.827392 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.827398 | controller | - ' ' 2026-03-20 16:26:25.827403 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.827409 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.827416 | controller | - ' ' 2026-03-20 16:26:25.827422 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.827428 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.827433 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.827439 | controller | - ' exit 1' 2026-03-20 16:26:25.827446 | controller | - ' fi' 2026-03-20 16:26:25.827452 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.827458 | controller | - ' ' 2026-03-20 16:26:25.827463 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.827481 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.827488 | controller | | base64 -d)' 2026-03-20 16:26:25.827495 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.827501 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.827507 | controller | - ' exit 1' 2026-03-20 16:26:25.827513 | controller | - ' fi' 2026-03-20 16:26:25.827518 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.827524 | controller | - ' ' 2026-03-20 16:26:25.827530 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.827535 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.827541 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.827547 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.827553 | controller | --timeout=300s' 2026-03-20 16:26:25.827558 | controller | - ' ' 2026-03-20 16:26:25.827567 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.827573 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.827579 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.827585 | controller | - ' ' 2026-03-20 16:26:25.827591 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.827597 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.827603 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.827609 | controller | - ' ' 2026-03-20 16:26:25.827615 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.827620 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.827626 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.827632 | controller | available"' 2026-03-20 16:26:25.827638 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.827647 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.827653 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.827659 | controller | - ' exit 1' 2026-03-20 16:26:25.827665 | controller | - ' fi' 2026-03-20 16:26:25.827671 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.827677 | controller | - ' ' 2026-03-20 16:26:25.827683 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.827689 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.827722 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.827728 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.827766 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.827773 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.827779 | controller | -- \' 2026-03-20 16:26:25.827785 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.827791 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.827797 | controller | - ' logger.go:42: 16:17:03 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.827803 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.827809 | controller | - ' logger.go:42: 16:17:03 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.827815 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.827821 | controller | found' 2026-03-20 16:26:25.827828 | controller | - ' logger.go:42: 16:17:04 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.827834 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.827839 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.827846 | controller | - ' ' 2026-03-20 16:26:25.827852 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.827858 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.827865 | controller | - ' ' 2026-03-20 16:26:25.827871 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.827881 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.827887 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.827892 | controller | - ' exit 1' 2026-03-20 16:26:25.827898 | controller | - ' fi' 2026-03-20 16:26:25.827904 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.827913 | controller | - ' ' 2026-03-20 16:26:25.827919 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.827925 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.827931 | controller | | base64 -d)' 2026-03-20 16:26:25.827950 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.827956 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.827962 | controller | - ' exit 1' 2026-03-20 16:26:25.827968 | controller | - ' fi' 2026-03-20 16:26:25.827974 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.827981 | controller | - ' ' 2026-03-20 16:26:25.827987 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.827993 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.827999 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.828005 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.828011 | controller | --timeout=300s' 2026-03-20 16:26:25.828017 | controller | - ' ' 2026-03-20 16:26:25.828023 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.828029 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.828035 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.828040 | controller | - ' ' 2026-03-20 16:26:25.828047 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.828053 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.828059 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.828065 | controller | - ' ' 2026-03-20 16:26:25.828072 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.828078 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.828084 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.828090 | controller | available"' 2026-03-20 16:26:25.828096 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.828102 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.828109 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.828114 | controller | - ' exit 1' 2026-03-20 16:26:25.828126 | controller | - ' fi' 2026-03-20 16:26:25.828133 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.828139 | controller | - ' ' 2026-03-20 16:26:25.828145 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.828151 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.828158 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.828164 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.828173 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.828179 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.828186 | controller | -- \' 2026-03-20 16:26:25.828192 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.828198 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.828205 | controller | - ' logger.go:42: 16:17:04 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.828211 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.828217 | controller | - ' logger.go:42: 16:17:04 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.828223 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.828228 | controller | found' 2026-03-20 16:26:25.828234 | controller | - ' logger.go:42: 16:17:05 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.828240 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.828247 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.828253 | controller | - ' ' 2026-03-20 16:26:25.828259 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.828265 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.828271 | controller | - ' ' 2026-03-20 16:26:25.828277 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.828282 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.828289 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.828295 | controller | - ' exit 1' 2026-03-20 16:26:25.828301 | controller | - ' fi' 2026-03-20 16:26:25.828307 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.828312 | controller | - ' ' 2026-03-20 16:26:25.828318 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.828324 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.828331 | controller | | base64 -d)' 2026-03-20 16:26:25.828337 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.828355 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.828362 | controller | - ' exit 1' 2026-03-20 16:26:25.828368 | controller | - ' fi' 2026-03-20 16:26:25.828374 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.828380 | controller | - ' ' 2026-03-20 16:26:25.828387 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.828392 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.828398 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.828404 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.828410 | controller | --timeout=300s' 2026-03-20 16:26:25.828416 | controller | - ' ' 2026-03-20 16:26:25.828422 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.828428 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.828434 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.828440 | controller | - ' ' 2026-03-20 16:26:25.828447 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.828457 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.828463 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.828469 | controller | - ' ' 2026-03-20 16:26:25.828476 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.828482 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.828488 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.828494 | controller | available"' 2026-03-20 16:26:25.828501 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.828506 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.828512 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.828518 | controller | - ' exit 1' 2026-03-20 16:26:25.828524 | controller | - ' fi' 2026-03-20 16:26:25.828530 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.828536 | controller | - ' ' 2026-03-20 16:26:25.828546 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.828552 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.828558 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.828563 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.828569 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.828575 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.828581 | controller | -- \' 2026-03-20 16:26:25.828587 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.828593 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.828599 | controller | - ' logger.go:42: 16:17:05 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.828605 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.828611 | controller | - ' logger.go:42: 16:17:05 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.828617 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.828623 | controller | found' 2026-03-20 16:26:25.828629 | controller | - ' logger.go:42: 16:17:06 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.828635 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.828641 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.828647 | controller | - ' ' 2026-03-20 16:26:25.828653 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.828659 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.828664 | controller | - ' ' 2026-03-20 16:26:25.828670 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.828676 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.828681 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.828691 | controller | - ' exit 1' 2026-03-20 16:26:25.828718 | controller | - ' fi' 2026-03-20 16:26:25.828724 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.828730 | controller | - ' ' 2026-03-20 16:26:25.828735 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.828741 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.828754 | controller | | base64 -d)' 2026-03-20 16:26:25.828760 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.828778 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.828784 | controller | - ' exit 1' 2026-03-20 16:26:25.828791 | controller | - ' fi' 2026-03-20 16:26:25.828796 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.828802 | controller | - ' ' 2026-03-20 16:26:25.828808 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.828814 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.828823 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.828830 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.828836 | controller | --timeout=300s' 2026-03-20 16:26:25.828841 | controller | - ' ' 2026-03-20 16:26:25.828847 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.828852 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.828858 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.828863 | controller | - ' ' 2026-03-20 16:26:25.828869 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.828875 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.828881 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.828886 | controller | - ' ' 2026-03-20 16:26:25.828892 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.828898 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.828904 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.828910 | controller | available"' 2026-03-20 16:26:25.828916 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.828922 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.828927 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.828933 | controller | - ' exit 1' 2026-03-20 16:26:25.828939 | controller | - ' fi' 2026-03-20 16:26:25.828945 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.828951 | controller | - ' ' 2026-03-20 16:26:25.828957 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.828963 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.828969 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.828975 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.828981 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.828987 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.828993 | controller | -- \' 2026-03-20 16:26:25.828998 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.829004 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.829015 | controller | - ' logger.go:42: 16:17:06 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.829021 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.829031 | controller | - ' logger.go:42: 16:17:07 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.829037 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.829043 | controller | found' 2026-03-20 16:26:25.829049 | controller | - ' logger.go:42: 16:17:08 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.829055 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.829061 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.829067 | controller | - ' ' 2026-03-20 16:26:25.829073 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.829079 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.829084 | controller | - ' ' 2026-03-20 16:26:25.829090 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.829096 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.829101 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.829106 | controller | - ' exit 1' 2026-03-20 16:26:25.829110 | controller | - ' fi' 2026-03-20 16:26:25.829114 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.829118 | controller | - ' ' 2026-03-20 16:26:25.829123 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.829127 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.829131 | controller | | base64 -d)' 2026-03-20 16:26:25.829135 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.829139 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.829144 | controller | - ' exit 1' 2026-03-20 16:26:25.829159 | controller | - ' fi' 2026-03-20 16:26:25.829164 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.829168 | controller | - ' ' 2026-03-20 16:26:25.829173 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.829177 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.829181 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.829185 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.829189 | controller | --timeout=300s' 2026-03-20 16:26:25.829193 | controller | - ' ' 2026-03-20 16:26:25.829198 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.829202 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.829206 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.829210 | controller | - ' ' 2026-03-20 16:26:25.829214 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.829221 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.829226 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.829230 | controller | - ' ' 2026-03-20 16:26:25.829234 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.829238 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.829242 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.829249 | controller | available"' 2026-03-20 16:26:25.829253 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.829257 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.829261 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.829265 | controller | - ' exit 1' 2026-03-20 16:26:25.829270 | controller | - ' fi' 2026-03-20 16:26:25.829274 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.829278 | controller | - ' ' 2026-03-20 16:26:25.829282 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.829286 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.829291 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.829295 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.829299 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.829303 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.829307 | controller | -- \' 2026-03-20 16:26:25.829311 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.829315 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.829320 | controller | - ' logger.go:42: 16:17:08 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.829324 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.829328 | controller | - ' logger.go:42: 16:17:08 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.829332 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.829336 | controller | found' 2026-03-20 16:26:25.829340 | controller | - ' logger.go:42: 16:17:09 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.829345 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.829349 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.829353 | controller | - ' ' 2026-03-20 16:26:25.829357 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.829361 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.829365 | controller | - ' ' 2026-03-20 16:26:25.829370 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.829374 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.829378 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.829382 | controller | - ' exit 1' 2026-03-20 16:26:25.829386 | controller | - ' fi' 2026-03-20 16:26:25.829391 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.829395 | controller | - ' ' 2026-03-20 16:26:25.829401 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.829405 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.829409 | controller | | base64 -d)' 2026-03-20 16:26:25.829414 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.829419 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.829424 | controller | - ' exit 1' 2026-03-20 16:26:25.829430 | controller | - ' fi' 2026-03-20 16:26:25.829446 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.829453 | controller | - ' ' 2026-03-20 16:26:25.829460 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.829464 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.829468 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.829475 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.829479 | controller | --timeout=300s' 2026-03-20 16:26:25.829485 | controller | - ' ' 2026-03-20 16:26:25.829490 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.829495 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.829499 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.829503 | controller | - ' ' 2026-03-20 16:26:25.829507 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.829511 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.829516 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.829520 | controller | - ' ' 2026-03-20 16:26:25.829524 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.829528 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.829533 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.829537 | controller | available"' 2026-03-20 16:26:25.829541 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.829545 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.829549 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.829554 | controller | - ' exit 1' 2026-03-20 16:26:25.829558 | controller | - ' fi' 2026-03-20 16:26:25.829562 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.829566 | controller | - ' ' 2026-03-20 16:26:25.829570 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.829575 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.829579 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.829583 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.829587 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.829592 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.829596 | controller | -- \' 2026-03-20 16:26:25.829600 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.829604 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.829609 | controller | - ' logger.go:42: 16:17:09 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.829613 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.829617 | controller | - ' logger.go:42: 16:17:09 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.829621 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.829625 | controller | found' 2026-03-20 16:26:25.829630 | controller | - ' logger.go:42: 16:17:10 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.829637 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.829642 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.829646 | controller | - ' ' 2026-03-20 16:26:25.829650 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.829654 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.829659 | controller | - ' ' 2026-03-20 16:26:25.829663 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.829667 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.829671 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.829676 | controller | - ' exit 1' 2026-03-20 16:26:25.829680 | controller | - ' fi' 2026-03-20 16:26:25.829684 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.829688 | controller | - ' ' 2026-03-20 16:26:25.829711 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.829721 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.829727 | controller | | base64 -d)' 2026-03-20 16:26:25.829733 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.829739 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.829745 | controller | - ' exit 1' 2026-03-20 16:26:25.829751 | controller | - ' fi' 2026-03-20 16:26:25.829757 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.829762 | controller | - ' ' 2026-03-20 16:26:25.829780 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.829787 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.829793 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.829798 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.829804 | controller | --timeout=300s' 2026-03-20 16:26:25.829810 | controller | - ' ' 2026-03-20 16:26:25.829816 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.829821 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.829827 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.829833 | controller | - ' ' 2026-03-20 16:26:25.829839 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.829845 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.829851 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.829857 | controller | - ' ' 2026-03-20 16:26:25.829863 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.829868 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.829874 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.829880 | controller | available"' 2026-03-20 16:26:25.829886 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.829891 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.829898 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.829903 | controller | - ' exit 1' 2026-03-20 16:26:25.829909 | controller | - ' fi' 2026-03-20 16:26:25.829915 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.829925 | controller | - ' ' 2026-03-20 16:26:25.829931 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.829937 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.829943 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.829949 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.829955 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.829960 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.829966 | controller | -- \' 2026-03-20 16:26:25.829972 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.829978 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.829984 | controller | - ' logger.go:42: 16:17:10 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.829990 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.829996 | controller | - ' logger.go:42: 16:17:10 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.830002 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.830008 | controller | found' 2026-03-20 16:26:25.830013 | controller | - ' logger.go:42: 16:17:11 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.830019 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.830025 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.830031 | controller | - ' ' 2026-03-20 16:26:25.830037 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.830043 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.830049 | controller | - ' ' 2026-03-20 16:26:25.830054 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.830060 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.830066 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.830072 | controller | - ' exit 1' 2026-03-20 16:26:25.830078 | controller | - ' fi' 2026-03-20 16:26:25.830084 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.830090 | controller | - ' ' 2026-03-20 16:26:25.830095 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.830101 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.830107 | controller | | base64 -d)' 2026-03-20 16:26:25.830113 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.830119 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.830124 | controller | - ' exit 1' 2026-03-20 16:26:25.830130 | controller | - ' fi' 2026-03-20 16:26:25.830135 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.830141 | controller | - ' ' 2026-03-20 16:26:25.830146 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.830162 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.830168 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.830173 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.830178 | controller | --timeout=300s' 2026-03-20 16:26:25.830184 | controller | - ' ' 2026-03-20 16:26:25.830189 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.830197 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.830203 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.830208 | controller | - ' ' 2026-03-20 16:26:25.830213 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.830218 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.830223 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.830228 | controller | - ' ' 2026-03-20 16:26:25.830236 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.830242 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.830248 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.830254 | controller | available"' 2026-03-20 16:26:25.830259 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.830265 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.830270 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.830275 | controller | - ' exit 1' 2026-03-20 16:26:25.830280 | controller | - ' fi' 2026-03-20 16:26:25.830285 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.830290 | controller | - ' ' 2026-03-20 16:26:25.830295 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.830301 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.830308 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.830313 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.830319 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.830326 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.830335 | controller | -- \' 2026-03-20 16:26:25.830341 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.830347 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.830353 | controller | - ' logger.go:42: 16:17:11 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.830358 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.830364 | controller | - ' logger.go:42: 16:17:11 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.830370 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.830375 | controller | found' 2026-03-20 16:26:25.830381 | controller | - ' logger.go:42: 16:17:12 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.830386 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.830392 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.830397 | controller | - ' ' 2026-03-20 16:26:25.830403 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.830409 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.830415 | controller | - ' ' 2026-03-20 16:26:25.830421 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.830430 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.830437 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.830443 | controller | - ' exit 1' 2026-03-20 16:26:25.830449 | controller | - ' fi' 2026-03-20 16:26:25.830455 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.830460 | controller | - ' ' 2026-03-20 16:26:25.830466 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.830472 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.830478 | controller | | base64 -d)' 2026-03-20 16:26:25.830484 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.830490 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.830496 | controller | - ' exit 1' 2026-03-20 16:26:25.830502 | controller | - ' fi' 2026-03-20 16:26:25.830508 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.830514 | controller | - ' ' 2026-03-20 16:26:25.830519 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.830538 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.830545 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.830550 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.830556 | controller | --timeout=300s' 2026-03-20 16:26:25.830562 | controller | - ' ' 2026-03-20 16:26:25.830568 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.830574 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.830580 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.830586 | controller | - ' ' 2026-03-20 16:26:25.830590 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.830594 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.830599 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.830603 | controller | - ' ' 2026-03-20 16:26:25.830607 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.830611 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.830616 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.830620 | controller | available"' 2026-03-20 16:26:25.830624 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.830628 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.830632 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.830637 | controller | - ' exit 1' 2026-03-20 16:26:25.830641 | controller | - ' fi' 2026-03-20 16:26:25.830645 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.830651 | controller | - ' ' 2026-03-20 16:26:25.830657 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.830663 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.830669 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.830675 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.830681 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.830691 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.830715 | controller | -- \' 2026-03-20 16:26:25.830719 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.830724 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.830728 | controller | - ' logger.go:42: 16:17:12 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.830732 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.830736 | controller | - ' logger.go:42: 16:17:12 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.830740 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.830745 | controller | found' 2026-03-20 16:26:25.830749 | controller | - ' logger.go:42: 16:17:13 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.830753 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.830757 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.830761 | controller | - ' ' 2026-03-20 16:26:25.830766 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.830770 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.830774 | controller | - ' ' 2026-03-20 16:26:25.830778 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.830782 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.830786 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.830791 | controller | - ' exit 1' 2026-03-20 16:26:25.830795 | controller | - ' fi' 2026-03-20 16:26:25.830799 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.830803 | controller | - ' ' 2026-03-20 16:26:25.830810 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.830815 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.830819 | controller | | base64 -d)' 2026-03-20 16:26:25.830823 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.830827 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.830831 | controller | - ' exit 1' 2026-03-20 16:26:25.830836 | controller | - ' fi' 2026-03-20 16:26:25.830840 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.830844 | controller | - ' ' 2026-03-20 16:26:25.830848 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.830852 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.830868 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.830875 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.830881 | controller | --timeout=300s' 2026-03-20 16:26:25.830887 | controller | - ' ' 2026-03-20 16:26:25.830892 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.830898 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.830904 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.830910 | controller | - ' ' 2026-03-20 16:26:25.830916 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.830921 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.830931 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.830936 | controller | - ' ' 2026-03-20 16:26:25.830942 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.830948 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.830957 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.830963 | controller | available"' 2026-03-20 16:26:25.830969 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.830977 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.830983 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.830989 | controller | - ' exit 1' 2026-03-20 16:26:25.830995 | controller | - ' fi' 2026-03-20 16:26:25.831001 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.831007 | controller | - ' ' 2026-03-20 16:26:25.831012 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.831018 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.831024 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.831029 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.831035 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.831041 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.831047 | controller | -- \' 2026-03-20 16:26:25.831053 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.831058 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.831064 | controller | - ' logger.go:42: 16:17:13 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.831070 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.831076 | controller | - ' logger.go:42: 16:17:14 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.831080 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.831084 | controller | found' 2026-03-20 16:26:25.831089 | controller | - ' logger.go:42: 16:17:15 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.831093 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.831097 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.831101 | controller | - ' ' 2026-03-20 16:26:25.831105 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.831110 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.831114 | controller | - ' ' 2026-03-20 16:26:25.831118 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.831122 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.831126 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.831131 | controller | - ' exit 1' 2026-03-20 16:26:25.831135 | controller | - ' fi' 2026-03-20 16:26:25.831139 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.831143 | controller | - ' ' 2026-03-20 16:26:25.831147 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.831151 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.831178 | controller | | base64 -d)' 2026-03-20 16:26:25.831182 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.831187 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.831191 | controller | - ' exit 1' 2026-03-20 16:26:25.831195 | controller | - ' fi' 2026-03-20 16:26:25.831199 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.831203 | controller | - ' ' 2026-03-20 16:26:25.831208 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.831212 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.831226 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.831231 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.831235 | controller | --timeout=300s' 2026-03-20 16:26:25.831239 | controller | - ' ' 2026-03-20 16:26:25.831244 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.831248 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.831252 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.831256 | controller | - ' ' 2026-03-20 16:26:25.831261 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.831266 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.831272 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.831278 | controller | - ' ' 2026-03-20 16:26:25.831284 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.831289 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.831293 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.831297 | controller | available"' 2026-03-20 16:26:25.831302 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.831306 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.831310 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.831314 | controller | - ' exit 1' 2026-03-20 16:26:25.831318 | controller | - ' fi' 2026-03-20 16:26:25.831323 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.831327 | controller | - ' ' 2026-03-20 16:26:25.831331 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.831335 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.831339 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.831343 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.831348 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.831352 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.831356 | controller | -- \' 2026-03-20 16:26:25.831361 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.831367 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.831373 | controller | - ' logger.go:42: 16:17:15 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.831382 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.831387 | controller | - ' logger.go:42: 16:17:15 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.831393 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.831399 | controller | found' 2026-03-20 16:26:25.831405 | controller | - ' logger.go:42: 16:17:16 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.831411 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.831420 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.831426 | controller | - ' ' 2026-03-20 16:26:25.831432 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.831438 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.831444 | controller | - ' ' 2026-03-20 16:26:25.831450 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.831456 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.831462 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.831467 | controller | - ' exit 1' 2026-03-20 16:26:25.831473 | controller | - ' fi' 2026-03-20 16:26:25.831479 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.831485 | controller | - ' ' 2026-03-20 16:26:25.831491 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.831496 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.831502 | controller | | base64 -d)' 2026-03-20 16:26:25.831508 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.831514 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.831520 | controller | - ' exit 1' 2026-03-20 16:26:25.831526 | controller | - ' fi' 2026-03-20 16:26:25.831532 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.831538 | controller | - ' ' 2026-03-20 16:26:25.831544 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.831550 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.831555 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.831573 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.831580 | controller | --timeout=300s' 2026-03-20 16:26:25.831585 | controller | - ' ' 2026-03-20 16:26:25.831591 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.831597 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.831603 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.831609 | controller | - ' ' 2026-03-20 16:26:25.831615 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.831621 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.831633 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.831640 | controller | - ' ' 2026-03-20 16:26:25.831646 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.831651 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.831657 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.831667 | controller | available"' 2026-03-20 16:26:25.831673 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.831682 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.831688 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.831710 | controller | - ' exit 1' 2026-03-20 16:26:25.831720 | controller | - ' fi' 2026-03-20 16:26:25.831726 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.831732 | controller | - ' ' 2026-03-20 16:26:25.831737 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.831763 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.831770 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.831776 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.831781 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.831786 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.831792 | controller | -- \' 2026-03-20 16:26:25.831797 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.831803 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.831809 | controller | - ' logger.go:42: 16:17:16 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.831815 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.831821 | controller | - ' logger.go:42: 16:17:16 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.831827 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.831833 | controller | found' 2026-03-20 16:26:25.831838 | controller | - ' logger.go:42: 16:17:17 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.831844 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.831851 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.831857 | controller | - ' ' 2026-03-20 16:26:25.831863 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.831869 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.831875 | controller | - ' ' 2026-03-20 16:26:25.831880 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.831886 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.831892 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.831898 | controller | - ' exit 1' 2026-03-20 16:26:25.831904 | controller | - ' fi' 2026-03-20 16:26:25.831910 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.831916 | controller | - ' ' 2026-03-20 16:26:25.831922 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.831928 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.831933 | controller | | base64 -d)' 2026-03-20 16:26:25.831939 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.831945 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.831951 | controller | - ' exit 1' 2026-03-20 16:26:25.831956 | controller | - ' fi' 2026-03-20 16:26:25.831963 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.831974 | controller | - ' ' 2026-03-20 16:26:25.831980 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.831990 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.831996 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.832015 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.832023 | controller | --timeout=300s' 2026-03-20 16:26:25.832029 | controller | - ' ' 2026-03-20 16:26:25.832035 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.832041 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.832047 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.832053 | controller | - ' ' 2026-03-20 16:26:25.832059 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.832065 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.832071 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.832077 | controller | - ' ' 2026-03-20 16:26:25.832082 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.832088 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.832094 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.832100 | controller | available"' 2026-03-20 16:26:25.832106 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.832112 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.832118 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.832124 | controller | - ' exit 1' 2026-03-20 16:26:25.832130 | controller | - ' fi' 2026-03-20 16:26:25.832136 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.832142 | controller | - ' ' 2026-03-20 16:26:25.832152 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.832158 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.832163 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.832169 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.832175 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.832181 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.832187 | controller | -- \' 2026-03-20 16:26:25.832192 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.832196 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.832200 | controller | - ' logger.go:42: 16:17:17 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.832205 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.832209 | controller | - ' logger.go:42: 16:17:17 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.832213 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.832217 | controller | found' 2026-03-20 16:26:25.832221 | controller | - ' logger.go:42: 16:17:18 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.832226 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.832232 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.832237 | controller | - ' ' 2026-03-20 16:26:25.832241 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.832245 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.832249 | controller | - ' ' 2026-03-20 16:26:25.832254 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.832258 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.832262 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.832266 | controller | - ' exit 1' 2026-03-20 16:26:25.832271 | controller | - ' fi' 2026-03-20 16:26:25.832275 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.832279 | controller | - ' ' 2026-03-20 16:26:25.832283 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.832287 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.832292 | controller | | base64 -d)' 2026-03-20 16:26:25.832296 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.832300 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.832304 | controller | - ' exit 1' 2026-03-20 16:26:25.832308 | controller | - ' fi' 2026-03-20 16:26:25.832313 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.832317 | controller | - ' ' 2026-03-20 16:26:25.832321 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.832325 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.832329 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.832334 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.832338 | controller | --timeout=300s' 2026-03-20 16:26:25.832344 | controller | - ' ' 2026-03-20 16:26:25.832359 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.832365 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.832371 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.832377 | controller | - ' ' 2026-03-20 16:26:25.832383 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.832389 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.832394 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.832400 | controller | - ' ' 2026-03-20 16:26:25.832406 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.832412 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.832416 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.832421 | controller | available"' 2026-03-20 16:26:25.832425 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.832429 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.832433 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.832438 | controller | - ' exit 1' 2026-03-20 16:26:25.832442 | controller | - ' fi' 2026-03-20 16:26:25.832446 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.832450 | controller | - ' ' 2026-03-20 16:26:25.832457 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.832461 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.832466 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.832470 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.832474 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.832478 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.832482 | controller | -- \' 2026-03-20 16:26:25.832487 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.832491 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.832495 | controller | - ' logger.go:42: 16:17:18 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.832501 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.832510 | controller | - ' logger.go:42: 16:17:18 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.832515 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.832521 | controller | found' 2026-03-20 16:26:25.832527 | controller | - ' logger.go:42: 16:17:19 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.832532 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.832538 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.832544 | controller | - ' ' 2026-03-20 16:26:25.832550 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.832556 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.832561 | controller | - ' ' 2026-03-20 16:26:25.832566 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.832570 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.832574 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.832579 | controller | - ' exit 1' 2026-03-20 16:26:25.832583 | controller | - ' fi' 2026-03-20 16:26:25.832587 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.832591 | controller | - ' ' 2026-03-20 16:26:25.832598 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.832602 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.832607 | controller | | base64 -d)' 2026-03-20 16:26:25.832611 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.832615 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.832620 | controller | - ' exit 1' 2026-03-20 16:26:25.832625 | controller | - ' fi' 2026-03-20 16:26:25.832631 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.832637 | controller | - ' ' 2026-03-20 16:26:25.832643 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.832649 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.832655 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.832660 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.832666 | controller | --timeout=300s' 2026-03-20 16:26:25.832672 | controller | - ' ' 2026-03-20 16:26:25.832688 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.832711 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.832722 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.832728 | controller | - ' ' 2026-03-20 16:26:25.832734 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.832739 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.832745 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.832751 | controller | - ' ' 2026-03-20 16:26:25.832756 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.832762 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.832768 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.832774 | controller | available"' 2026-03-20 16:26:25.832780 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.832786 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.832791 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.832797 | controller | - ' exit 1' 2026-03-20 16:26:25.832803 | controller | - ' fi' 2026-03-20 16:26:25.832809 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.832815 | controller | - ' ' 2026-03-20 16:26:25.832820 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.832826 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.832832 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.832838 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.832843 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.832849 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.832855 | controller | -- \' 2026-03-20 16:26:25.832861 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.832867 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.832876 | controller | - ' logger.go:42: 16:17:19 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.832882 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.832888 | controller | - ' logger.go:42: 16:17:19 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.832893 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.832898 | controller | found' 2026-03-20 16:26:25.832904 | controller | - ' logger.go:42: 16:17:20 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.832908 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.832912 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.832916 | controller | - ' ' 2026-03-20 16:26:25.832921 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.832925 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.832929 | controller | - ' ' 2026-03-20 16:26:25.832933 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.832938 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.832942 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.832949 | controller | - ' exit 1' 2026-03-20 16:26:25.832953 | controller | - ' fi' 2026-03-20 16:26:25.832958 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.832962 | controller | - ' ' 2026-03-20 16:26:25.832966 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.832970 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.832974 | controller | | base64 -d)' 2026-03-20 16:26:25.832979 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.832984 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.832990 | controller | - ' exit 1' 2026-03-20 16:26:25.832996 | controller | - ' fi' 2026-03-20 16:26:25.833002 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.833008 | controller | - ' ' 2026-03-20 16:26:25.833014 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.833020 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.833026 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.833031 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.833037 | controller | --timeout=300s' 2026-03-20 16:26:25.833043 | controller | - ' ' 2026-03-20 16:26:25.833049 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.833066 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.833073 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.833079 | controller | - ' ' 2026-03-20 16:26:25.833085 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.833103 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.833110 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.833116 | controller | - ' ' 2026-03-20 16:26:25.833122 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.833128 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.833133 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.833139 | controller | available"' 2026-03-20 16:26:25.833145 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.833151 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.833156 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.833162 | controller | - ' exit 1' 2026-03-20 16:26:25.833168 | controller | - ' fi' 2026-03-20 16:26:25.833174 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.833180 | controller | - ' ' 2026-03-20 16:26:25.833186 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.833192 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.833197 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.833207 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.833213 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.833223 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.833229 | controller | -- \' 2026-03-20 16:26:25.833235 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.833240 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.833246 | controller | - ' logger.go:42: 16:17:20 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.833252 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.833258 | controller | - ' logger.go:42: 16:17:21 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.833264 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.833269 | controller | found' 2026-03-20 16:26:25.833273 | controller | - ' logger.go:42: 16:17:22 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.833277 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.833283 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.833292 | controller | - ' ' 2026-03-20 16:26:25.833298 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.833304 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.833310 | controller | - ' ' 2026-03-20 16:26:25.833315 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.833321 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.833327 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.833333 | controller | - ' exit 1' 2026-03-20 16:26:25.833339 | controller | - ' fi' 2026-03-20 16:26:25.833345 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.833351 | controller | - ' ' 2026-03-20 16:26:25.833356 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.833362 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.833368 | controller | | base64 -d)' 2026-03-20 16:26:25.833374 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.833379 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.833385 | controller | - ' exit 1' 2026-03-20 16:26:25.833391 | controller | - ' fi' 2026-03-20 16:26:25.833397 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.833402 | controller | - ' ' 2026-03-20 16:26:25.833408 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.833414 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.833419 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.833425 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.833431 | controller | --timeout=300s' 2026-03-20 16:26:25.833437 | controller | - ' ' 2026-03-20 16:26:25.833443 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.833459 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.833466 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.833471 | controller | - ' ' 2026-03-20 16:26:25.833480 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.833486 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.833495 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.833501 | controller | - ' ' 2026-03-20 16:26:25.833507 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.833512 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.833518 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.833524 | controller | available"' 2026-03-20 16:26:25.833530 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.833535 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.833541 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.833547 | controller | - ' exit 1' 2026-03-20 16:26:25.833553 | controller | - ' fi' 2026-03-20 16:26:25.833559 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.833564 | controller | - ' ' 2026-03-20 16:26:25.833570 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.833576 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.833582 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.833588 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.833594 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.833599 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.833605 | controller | -- \' 2026-03-20 16:26:25.833611 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.833617 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.833623 | controller | - ' logger.go:42: 16:17:22 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.833628 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.833634 | controller | - ' logger.go:42: 16:17:22 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.833640 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.833646 | controller | found' 2026-03-20 16:26:25.833652 | controller | - ' logger.go:42: 16:17:23 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.833657 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.833663 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.833669 | controller | - ' ' 2026-03-20 16:26:25.833675 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.833680 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.833686 | controller | - ' ' 2026-03-20 16:26:25.833706 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.833716 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.833722 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.833730 | controller | - ' exit 1' 2026-03-20 16:26:25.833735 | controller | - ' fi' 2026-03-20 16:26:25.833739 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.833743 | controller | - ' ' 2026-03-20 16:26:25.833747 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.833752 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.833756 | controller | | base64 -d)' 2026-03-20 16:26:25.833763 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.833767 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.833771 | controller | - ' exit 1' 2026-03-20 16:26:25.833775 | controller | - ' fi' 2026-03-20 16:26:25.833779 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.833784 | controller | - ' ' 2026-03-20 16:26:25.833788 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.833792 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.833796 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.833800 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.833805 | controller | --timeout=300s' 2026-03-20 16:26:25.833809 | controller | - ' ' 2026-03-20 16:26:25.833813 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.833826 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.833831 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.833835 | controller | - ' ' 2026-03-20 16:26:25.833840 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.833846 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.833852 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.833858 | controller | - ' ' 2026-03-20 16:26:25.833864 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.833869 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.833875 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.833881 | controller | available"' 2026-03-20 16:26:25.833886 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.833892 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.833898 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.833904 | controller | - ' exit 1' 2026-03-20 16:26:25.833910 | controller | - ' fi' 2026-03-20 16:26:25.833916 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.833921 | controller | - ' ' 2026-03-20 16:26:25.833927 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.833931 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.833936 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.833940 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.833944 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.833948 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.833953 | controller | -- \' 2026-03-20 16:26:25.833957 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.833964 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.833968 | controller | - ' logger.go:42: 16:17:23 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.833975 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.833981 | controller | - ' logger.go:42: 16:17:23 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.833987 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.833992 | controller | found' 2026-03-20 16:26:25.833998 | controller | - ' logger.go:42: 16:17:24 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.834004 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.834010 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.834016 | controller | - ' ' 2026-03-20 16:26:25.834021 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.834026 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.834030 | controller | - ' ' 2026-03-20 16:26:25.834034 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.834038 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.834043 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.834047 | controller | - ' exit 1' 2026-03-20 16:26:25.834051 | controller | - ' fi' 2026-03-20 16:26:25.834055 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.834059 | controller | - ' ' 2026-03-20 16:26:25.834064 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.834068 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.834072 | controller | | base64 -d)' 2026-03-20 16:26:25.834076 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.834080 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.834085 | controller | - ' exit 1' 2026-03-20 16:26:25.834089 | controller | - ' fi' 2026-03-20 16:26:25.834093 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.834097 | controller | - ' ' 2026-03-20 16:26:25.834101 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.834106 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.834110 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.834114 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.834118 | controller | --timeout=300s' 2026-03-20 16:26:25.834122 | controller | - ' ' 2026-03-20 16:26:25.834127 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.834131 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.834145 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.834149 | controller | - ' ' 2026-03-20 16:26:25.834154 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.834158 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.834162 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.834166 | controller | - ' ' 2026-03-20 16:26:25.834171 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.834178 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.834182 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.834186 | controller | available"' 2026-03-20 16:26:25.834190 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.834197 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.834201 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.834205 | controller | - ' exit 1' 2026-03-20 16:26:25.834209 | controller | - ' fi' 2026-03-20 16:26:25.834214 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.834220 | controller | - ' ' 2026-03-20 16:26:25.834224 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.834229 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.834233 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.834237 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.834241 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.834246 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.834250 | controller | -- \' 2026-03-20 16:26:25.834254 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.834258 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.834262 | controller | - ' logger.go:42: 16:17:24 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.834267 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.834271 | controller | - ' logger.go:42: 16:17:24 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.834275 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.834279 | controller | found' 2026-03-20 16:26:25.834284 | controller | - ' logger.go:42: 16:17:25 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.834288 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.834292 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.834298 | controller | - ' ' 2026-03-20 16:26:25.834304 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.834309 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.834315 | controller | - ' ' 2026-03-20 16:26:25.834320 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.834324 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.834329 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.834333 | controller | - ' exit 1' 2026-03-20 16:26:25.834337 | controller | - ' fi' 2026-03-20 16:26:25.834341 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.834346 | controller | - ' ' 2026-03-20 16:26:25.834350 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.834354 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.834358 | controller | | base64 -d)' 2026-03-20 16:26:25.834362 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.834369 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.834374 | controller | - ' exit 1' 2026-03-20 16:26:25.834379 | controller | - ' fi' 2026-03-20 16:26:25.834383 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.834387 | controller | - ' ' 2026-03-20 16:26:25.834391 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.834395 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.834401 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.834407 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.834412 | controller | --timeout=300s' 2026-03-20 16:26:25.834418 | controller | - ' ' 2026-03-20 16:26:25.834424 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.834430 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.834444 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.834449 | controller | - ' ' 2026-03-20 16:26:25.834454 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.834459 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.834463 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.834467 | controller | - ' ' 2026-03-20 16:26:25.834471 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.834475 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.834480 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.834484 | controller | available"' 2026-03-20 16:26:25.834488 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.834493 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.834499 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.834505 | controller | - ' exit 1' 2026-03-20 16:26:25.834511 | controller | - ' fi' 2026-03-20 16:26:25.834517 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.834523 | controller | - ' ' 2026-03-20 16:26:25.834529 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.834535 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.834540 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.834546 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.834552 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.834558 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.834579 | controller | -- \' 2026-03-20 16:26:25.834585 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.834592 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.834598 | controller | - ' logger.go:42: 16:17:25 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.834603 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.834614 | controller | - ' logger.go:42: 16:17:25 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.834620 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.834626 | controller | found' 2026-03-20 16:26:25.834631 | controller | - ' logger.go:42: 16:17:26 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.834637 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.834643 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.834652 | controller | - ' ' 2026-03-20 16:26:25.834658 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.834663 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.834669 | controller | - ' ' 2026-03-20 16:26:25.834675 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.834680 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.834686 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.834708 | controller | - ' exit 1' 2026-03-20 16:26:25.834718 | controller | - ' fi' 2026-03-20 16:26:25.834724 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.834730 | controller | - ' ' 2026-03-20 16:26:25.834736 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.834742 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.834747 | controller | | base64 -d)' 2026-03-20 16:26:25.834753 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.834759 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.834765 | controller | - ' exit 1' 2026-03-20 16:26:25.834776 | controller | - ' fi' 2026-03-20 16:26:25.834782 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.834788 | controller | - ' ' 2026-03-20 16:26:25.834794 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.834799 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.834805 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.834811 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.834816 | controller | --timeout=300s' 2026-03-20 16:26:25.834822 | controller | - ' ' 2026-03-20 16:26:25.834828 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.834834 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.834840 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.834846 | controller | - ' ' 2026-03-20 16:26:25.834863 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.834870 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.834876 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.834882 | controller | - ' ' 2026-03-20 16:26:25.834888 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.834894 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.834900 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.834907 | controller | available"' 2026-03-20 16:26:25.834913 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.834919 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.834925 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.834930 | controller | - ' exit 1' 2026-03-20 16:26:25.834936 | controller | - ' fi' 2026-03-20 16:26:25.834942 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.834947 | controller | - ' ' 2026-03-20 16:26:25.834953 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.834962 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.834968 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.834973 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.834979 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.834984 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.834990 | controller | -- \' 2026-03-20 16:26:25.834996 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.835002 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.835008 | controller | - ' logger.go:42: 16:17:26 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.835014 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.835020 | controller | - ' logger.go:42: 16:17:26 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.835026 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.835033 | controller | found' 2026-03-20 16:26:25.835039 | controller | - ' logger.go:42: 16:17:27 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.835045 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.835051 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.835057 | controller | - ' ' 2026-03-20 16:26:25.835063 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.835070 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.835076 | controller | - ' ' 2026-03-20 16:26:25.835082 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.835088 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.835094 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.835100 | controller | - ' exit 1' 2026-03-20 16:26:25.835106 | controller | - ' fi' 2026-03-20 16:26:25.835112 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.835117 | controller | - ' ' 2026-03-20 16:26:25.835123 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.835130 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.835136 | controller | | base64 -d)' 2026-03-20 16:26:25.835142 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.835148 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.835153 | controller | - ' exit 1' 2026-03-20 16:26:25.835160 | controller | - ' fi' 2026-03-20 16:26:25.835166 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.835172 | controller | - ' ' 2026-03-20 16:26:25.835177 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.835183 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.835189 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.835195 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.835201 | controller | --timeout=300s' 2026-03-20 16:26:25.835207 | controller | - ' ' 2026-03-20 16:26:25.835213 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.835219 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.835229 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.835236 | controller | - ' ' 2026-03-20 16:26:25.835254 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.835261 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.835267 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.835273 | controller | - ' ' 2026-03-20 16:26:25.835279 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.835285 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.835294 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.835300 | controller | available"' 2026-03-20 16:26:25.835306 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.835312 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.835319 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.835325 | controller | - ' exit 1' 2026-03-20 16:26:25.835331 | controller | - ' fi' 2026-03-20 16:26:25.835337 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.835343 | controller | - ' ' 2026-03-20 16:26:25.835349 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.835355 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.835361 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.835367 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.835373 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.835379 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.835386 | controller | -- \' 2026-03-20 16:26:25.835391 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.835397 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.835403 | controller | - ' logger.go:42: 16:17:27 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.835408 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.835414 | controller | - ' logger.go:42: 16:17:28 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.835434 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.835441 | controller | found' 2026-03-20 16:26:25.835447 | controller | - ' logger.go:42: 16:17:29 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.835453 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.835459 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.835465 | controller | - ' ' 2026-03-20 16:26:25.835471 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.835477 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.835483 | controller | - ' ' 2026-03-20 16:26:25.835489 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.835495 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.835501 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.835510 | controller | - ' exit 1' 2026-03-20 16:26:25.835516 | controller | - ' fi' 2026-03-20 16:26:25.835522 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.835528 | controller | - ' ' 2026-03-20 16:26:25.835534 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.835540 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.835546 | controller | | base64 -d)' 2026-03-20 16:26:25.835552 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.835558 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.835564 | controller | - ' exit 1' 2026-03-20 16:26:25.835570 | controller | - ' fi' 2026-03-20 16:26:25.835576 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.835582 | controller | - ' ' 2026-03-20 16:26:25.835588 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.835593 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.835599 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.835606 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.835612 | controller | --timeout=300s' 2026-03-20 16:26:25.835618 | controller | - ' ' 2026-03-20 16:26:25.835624 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.835630 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.835640 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.835647 | controller | - ' ' 2026-03-20 16:26:25.835653 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.835671 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.835678 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.835684 | controller | - ' ' 2026-03-20 16:26:25.835690 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.835717 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.835724 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.835730 | controller | available"' 2026-03-20 16:26:25.835736 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.835742 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.835777 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.835785 | controller | - ' exit 1' 2026-03-20 16:26:25.835791 | controller | - ' fi' 2026-03-20 16:26:25.835797 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.835803 | controller | - ' ' 2026-03-20 16:26:25.835809 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.835815 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.835821 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.835827 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.835834 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.835840 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.835850 | controller | -- \' 2026-03-20 16:26:25.835856 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.835861 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.835867 | controller | - ' logger.go:42: 16:17:29 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.835873 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.835880 | controller | - ' logger.go:42: 16:17:29 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.835886 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.835892 | controller | found' 2026-03-20 16:26:25.835898 | controller | - ' logger.go:42: 16:17:30 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.835904 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.835911 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.835916 | controller | - ' ' 2026-03-20 16:26:25.835922 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.835928 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.835934 | controller | - ' ' 2026-03-20 16:26:25.835940 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.835946 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.835952 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.835958 | controller | - ' exit 1' 2026-03-20 16:26:25.835965 | controller | - ' fi' 2026-03-20 16:26:25.835971 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.835977 | controller | - ' ' 2026-03-20 16:26:25.835988 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.835994 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.836000 | controller | | base64 -d)' 2026-03-20 16:26:25.836006 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.836012 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.836018 | controller | - ' exit 1' 2026-03-20 16:26:25.836024 | controller | - ' fi' 2026-03-20 16:26:25.836030 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.836036 | controller | - ' ' 2026-03-20 16:26:25.836042 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.836048 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.836055 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.836061 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.836066 | controller | --timeout=300s' 2026-03-20 16:26:25.836072 | controller | - ' ' 2026-03-20 16:26:25.836079 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.836085 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.836091 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.836097 | controller | - ' ' 2026-03-20 16:26:25.836103 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.836122 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.836130 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.836140 | controller | - ' ' 2026-03-20 16:26:25.836146 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.836152 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.836158 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.836164 | controller | available"' 2026-03-20 16:26:25.836169 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.836175 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.836181 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.836187 | controller | - ' exit 1' 2026-03-20 16:26:25.836193 | controller | - ' fi' 2026-03-20 16:26:25.836198 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.836204 | controller | - ' ' 2026-03-20 16:26:25.836210 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.836216 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.836221 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.836227 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.836233 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.836239 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.836245 | controller | -- \' 2026-03-20 16:26:25.836250 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.836256 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.836262 | controller | - ' logger.go:42: 16:17:30 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.836268 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.836274 | controller | - ' logger.go:42: 16:17:30 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.836280 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.836286 | controller | found' 2026-03-20 16:26:25.836295 | controller | - ' logger.go:42: 16:17:31 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.836301 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.836307 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.836312 | controller | - ' ' 2026-03-20 16:26:25.836318 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.836324 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.836330 | controller | - ' ' 2026-03-20 16:26:25.836339 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.836345 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.836350 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.836356 | controller | - ' exit 1' 2026-03-20 16:26:25.836362 | controller | - ' fi' 2026-03-20 16:26:25.836368 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.836374 | controller | - ' ' 2026-03-20 16:26:25.836380 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.836386 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.836392 | controller | | base64 -d)' 2026-03-20 16:26:25.836398 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.836408 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.836414 | controller | - ' exit 1' 2026-03-20 16:26:25.836420 | controller | - ' fi' 2026-03-20 16:26:25.836426 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.836432 | controller | - ' ' 2026-03-20 16:26:25.836438 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.836443 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.836449 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.836455 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.836461 | controller | --timeout=300s' 2026-03-20 16:26:25.836467 | controller | - ' ' 2026-03-20 16:26:25.836473 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.836479 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.836484 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.836490 | controller | - ' ' 2026-03-20 16:26:25.836496 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.836502 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.836520 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.836527 | controller | - ' ' 2026-03-20 16:26:25.836533 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.836539 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.836545 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.836551 | controller | available"' 2026-03-20 16:26:25.836557 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.836563 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.836569 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.836574 | controller | - ' exit 1' 2026-03-20 16:26:25.836580 | controller | - ' fi' 2026-03-20 16:26:25.836586 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.836592 | controller | - ' ' 2026-03-20 16:26:25.836598 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.836604 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.836610 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.836615 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.836621 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.836627 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.836633 | controller | -- \' 2026-03-20 16:26:25.836639 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.836645 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.836649 | controller | - ' logger.go:42: 16:17:31 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.836653 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.836661 | controller | - ' logger.go:42: 16:17:31 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.836665 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.836669 | controller | found' 2026-03-20 16:26:25.836673 | controller | - ' logger.go:42: 16:17:32 | appcred-tests/2-deploy-appcred | running command: 2026-03-20 16:26:25.836677 | controller | [sh -c set -euo pipefail' 2026-03-20 16:26:25.836682 | controller | - ' NS="${NAMESPACE}"' 2026-03-20 16:26:25.836688 | controller | - ' ' 2026-03-20 16:26:25.836715 | controller | - ' echo "Waiting for KeystoneApplicationCredential ac-watcher to be Ready..."' 2026-03-20 16:26:25.836726 | controller | - ' oc wait -n "${NS}" appcred/ac-watcher --for=condition=Ready --timeout=300s' 2026-03-20 16:26:25.836732 | controller | - ' ' 2026-03-20 16:26:25.836738 | controller | - ' ac_id=$(oc get -n "${NS}" appcred/ac-watcher -o jsonpath=''{.status.acID}'')' 2026-03-20 16:26:25.836744 | controller | - ' if [ -z "${ac_id}" ]; then' 2026-03-20 16:26:25.836750 | controller | - ' echo "ERROR: appcred/ac-watcher.status.acID is empty"' 2026-03-20 16:26:25.836755 | controller | - ' exit 1' 2026-03-20 16:26:25.836759 | controller | - ' fi' 2026-03-20 16:26:25.836763 | controller | - ' echo "ac-watcher.status.acID=${ac_id}"' 2026-03-20 16:26:25.836767 | controller | - ' ' 2026-03-20 16:26:25.836772 | controller | - ' oc get -n "${NS}" secret/ac-watcher-secret >/dev/null' 2026-03-20 16:26:25.836776 | controller | - ' secret_ac_id=$(oc get -n "${NS}" secret/ac-watcher-secret -o jsonpath=''{.data.AC_ID}'' 2026-03-20 16:26:25.836780 | controller | | base64 -d)' 2026-03-20 16:26:25.836784 | controller | - ' if [ "${secret_ac_id}" != "${ac_id}" ]; then' 2026-03-20 16:26:25.836788 | controller | - ' echo "ERROR: Secret AC_ID (${secret_ac_id}) != appcred.status.acID (${ac_id})"' 2026-03-20 16:26:25.836793 | controller | - ' exit 1' 2026-03-20 16:26:25.836797 | controller | - ' fi' 2026-03-20 16:26:25.836801 | controller | - ' echo "✓ Secret AC_ID matches appcred.status.acID"' 2026-03-20 16:26:25.836805 | controller | - ' ' 2026-03-20 16:26:25.836810 | controller | - ' echo "Waiting for watcher rollouts..."' 2026-03-20 16:26:25.836814 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-api --timeout=300s' 2026-03-20 16:26:25.836818 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-applier --timeout=300s' 2026-03-20 16:26:25.836822 | controller | - ' oc rollout status -n "${NS}" statefulset/watcher-kuttl-decision-engine 2026-03-20 16:26:25.836826 | controller | --timeout=300s' 2026-03-20 16:26:25.836831 | controller | - ' ' 2026-03-20 16:26:25.836835 | controller | - ' old_api_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.api_uid}'')' 2026-03-20 16:26:25.836839 | controller | - ' old_applier_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.applier_uid}'')' 2026-03-20 16:26:25.836843 | controller | - ' old_de_uid=$(oc get -n "${NS}" configmap/appcred-watcher-pre -o jsonpath=''{.data.decision_engine_uid}'')' 2026-03-20 16:26:25.836848 | controller | - ' ' 2026-03-20 16:26:25.836852 | controller | - ' new_api_uid=$(oc get pod -n "${NS}" watcher-kuttl-api-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.836856 | controller | - ' new_applier_uid=$(oc get pod -n "${NS}" watcher-kuttl-applier-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.836870 | controller | - ' new_de_uid=$(oc get pod -n "${NS}" watcher-kuttl-decision-engine-0 -o jsonpath=''{.metadata.uid}'')' 2026-03-20 16:26:25.836874 | controller | - ' ' 2026-03-20 16:26:25.836879 | controller | - ' if [ "${old_api_uid}" = "${new_api_uid}" ] || [ "${old_applier_uid}" = 2026-03-20 16:26:25.836884 | controller | "${new_applier_uid}" ] || [ "${old_de_uid}" = "${new_de_uid}" ]; then' 2026-03-20 16:26:25.836888 | controller | - ' echo "ERROR: expected watcher pods to restart after appcred secret became 2026-03-20 16:26:25.836893 | controller | available"' 2026-03-20 16:26:25.836897 | controller | - ' echo " api: ${old_api_uid} -> ${new_api_uid}"' 2026-03-20 16:26:25.836904 | controller | - ' echo " applier: ${old_applier_uid} -> ${new_applier_uid}"' 2026-03-20 16:26:25.836908 | controller | - ' echo " decision-engine: ${old_de_uid} -> ${new_de_uid}"' 2026-03-20 16:26:25.836912 | controller | - ' exit 1' 2026-03-20 16:26:25.836916 | controller | - ' fi' 2026-03-20 16:26:25.836921 | controller | - ' echo "✓ watcher pods restarted after appcred secret became available"' 2026-03-20 16:26:25.836925 | controller | - ' ' 2026-03-20 16:26:25.836929 | controller | - ' echo "Checking watcher config contains application_credential_id..."' 2026-03-20 16:26:25.836933 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-api-0 -c watcher-api -- \' 2026-03-20 16:26:25.836938 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.836942 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-applier-0 -c watcher-applier -- \' 2026-03-20 16:26:25.836951 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.836955 | controller | - ' oc exec -n "${NS}" pod/watcher-kuttl-decision-engine-0 -c watcher-decision-engine 2026-03-20 16:26:25.836959 | controller | -- \' 2026-03-20 16:26:25.836964 | controller | - ' bash -c "grep -q \"^application_credential_id = ${ac_id}$\" /etc/watcher/watcher.conf.d/00-default.conf"' 2026-03-20 16:26:25.836968 | controller | - ' echo "✓ watcher config contains expected application_credential_id"]' 2026-03-20 16:26:25.836972 | controller | - ' logger.go:42: 16:17:32 | appcred-tests/2-deploy-appcred | Waiting for KeystoneApplicationCredential 2026-03-20 16:26:25.836976 | controller | ac-watcher to be Ready...' 2026-03-20 16:26:25.836981 | controller | - ' logger.go:42: 16:17:32 | appcred-tests/2-deploy-appcred | Error from server 2026-03-20 16:26:25.836985 | controller | (NotFound): keystoneapplicationcredentials.keystone.openstack.org "ac-watcher" not 2026-03-20 16:26:25.836989 | controller | found' 2026-03-20 16:26:25.836993 | controller | - ' logger.go:42: 16:17:33 | appcred-tests/2-deploy-appcred | test step failed 2026-03-20 16:26:25.836997 | controller | 2-deploy-appcred' 2026-03-20 16:26:25.837002 | controller | - ' case.go:396: failed in step 2-deploy-appcred' 2026-03-20 16:26:25.837008 | controller | - ' case.go:398: command "NS=\"${NAMESPACE}\"\\n echo \"Waiting for KeystoneApplicationCredential..." 2026-03-20 16:26:25.837012 | controller | failed, exit status 1' 2026-03-20 16:26:25.837016 | controller | - ' logger.go:42: 16:17:33 | appcred-tests | skipping kubernetes event logging' 2026-03-20 16:26:25.837020 | controller | - === CONT kuttl/harness/watcher-notification 2026-03-20 16:26:25.837025 | controller | - ' logger.go:42: 16:17:33 | watcher-notification | Skipping creation of user-supplied 2026-03-20 16:26:25.837029 | controller | namespace: watcher-kuttl-default' 2026-03-20 16:26:25.837033 | controller | - ' logger.go:42: 16:17:33 | watcher-notification/0-cleanup-watcher | starting 2026-03-20 16:26:25.837038 | controller | test step 0-cleanup-watcher' 2026-03-20 16:26:25.837042 | controller | - ' logger.go:42: 16:17:33 | watcher-notification/0-cleanup-watcher | test step 2026-03-20 16:26:25.837047 | controller | completed 0-cleanup-watcher' 2026-03-20 16:26:25.837053 | controller | - ' logger.go:42: 16:17:33 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837059 | controller | starting test step 1-deploy-with-notification' 2026-03-20 16:26:25.837065 | controller | - ' logger.go:42: 16:17:33 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837070 | controller | Watcher:watcher-kuttl-default/watcher-kuttl created' 2026-03-20 16:26:25.837076 | controller | - ' logger.go:42: 16:17:33 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837081 | controller | running command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.837085 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.837089 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.837096 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.837102 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.837106 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.837112 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-20 16:26:25.837118 | controller | == 1 ]' 2026-03-20 16:26:25.837124 | controller | - ' else' 2026-03-20 16:26:25.837130 | controller | - ' exit 1' 2026-03-20 16:26:25.837136 | controller | - ' fi' 2026-03-20 16:26:25.837141 | controller | - ' ]' 2026-03-20 16:26:25.837147 | controller | - ' logger.go:42: 16:17:33 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837151 | controller | + oc project watcher-kuttl-default' 2026-03-20 16:26:25.837155 | controller | - ' logger.go:42: 16:17:34 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837160 | controller | Now using project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.837164 | controller | - ' logger.go:42: 16:17:34 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837177 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.837182 | controller | - ' logger.go:42: 16:17:34 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837186 | controller | ++ head -1' 2026-03-20 16:26:25.837191 | controller | - ' logger.go:42: 16:17:34 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837195 | controller | ++ grep -v ''^$''' 2026-03-20 16:26:25.837199 | controller | - ' logger.go:42: 16:17:34 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837203 | controller | + APIPOD=watcher-kuttl-api-0' 2026-03-20 16:26:25.837208 | controller | - ' logger.go:42: 16:17:34 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837212 | controller | + ''['' -n watcher-kuttl-api-0 '']''' 2026-03-20 16:26:25.837216 | controller | - ' logger.go:42: 16:17:34 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837220 | controller | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.837225 | controller | - ' logger.go:42: 16:17:34 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837229 | controller | ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+=''' 2026-03-20 16:26:25.837233 | controller | - ' logger.go:42: 16:17:35 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837237 | controller | error: Internal error occurred: error executing command in container: container 2026-03-20 16:26:25.837241 | controller | is not created or running' 2026-03-20 16:26:25.837246 | controller | - ' logger.go:42: 16:17:35 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837250 | controller | ++ echo' 2026-03-20 16:26:25.837254 | controller | - ' logger.go:42: 16:17:35 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837258 | controller | + ''['' 0 == 1 '']''' 2026-03-20 16:26:25.837263 | controller | - ' logger.go:42: 16:17:36 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837267 | controller | running command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.837271 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.837275 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.837279 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.837284 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.837288 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.837292 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-20 16:26:25.837296 | controller | == 1 ]' 2026-03-20 16:26:25.837301 | controller | - ' else' 2026-03-20 16:26:25.837305 | controller | - ' exit 1' 2026-03-20 16:26:25.837309 | controller | - ' fi' 2026-03-20 16:26:25.837316 | controller | - ' ]' 2026-03-20 16:26:25.837320 | controller | - ' logger.go:42: 16:17:36 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837324 | controller | + oc project watcher-kuttl-default' 2026-03-20 16:26:25.837328 | controller | - ' logger.go:42: 16:17:36 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837332 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.837336 | controller | - ' logger.go:42: 16:17:36 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837341 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.837345 | controller | - ' logger.go:42: 16:17:36 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837349 | controller | ++ head -1' 2026-03-20 16:26:25.837353 | controller | - ' logger.go:42: 16:17:36 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837357 | controller | ++ grep -v ''^$''' 2026-03-20 16:26:25.837362 | controller | - ' logger.go:42: 16:17:36 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837366 | controller | + APIPOD=' 2026-03-20 16:26:25.837372 | controller | - ' logger.go:42: 16:17:37 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837377 | controller | running command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.837381 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.837385 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.837389 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.837394 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.837398 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.837402 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-20 16:26:25.837406 | controller | == 1 ]' 2026-03-20 16:26:25.837411 | controller | - ' else' 2026-03-20 16:26:25.837415 | controller | - ' exit 1' 2026-03-20 16:26:25.837419 | controller | - ' fi' 2026-03-20 16:26:25.837423 | controller | - ' ]' 2026-03-20 16:26:25.837427 | controller | - ' logger.go:42: 16:17:37 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837432 | controller | + oc project watcher-kuttl-default' 2026-03-20 16:26:25.837438 | controller | - ' logger.go:42: 16:17:37 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837444 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.837448 | controller | - ' logger.go:42: 16:17:37 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837452 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.837457 | controller | - ' logger.go:42: 16:17:37 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837461 | controller | ++ head -1' 2026-03-20 16:26:25.837473 | controller | - ' logger.go:42: 16:17:37 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837478 | controller | ++ grep -v ''^$''' 2026-03-20 16:26:25.837482 | controller | - ' logger.go:42: 16:17:37 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837486 | controller | + APIPOD=' 2026-03-20 16:26:25.837491 | controller | - ' logger.go:42: 16:17:39 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837495 | controller | running command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.837499 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.837503 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.837508 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.837512 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.837516 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.837523 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-20 16:26:25.837528 | controller | == 1 ]' 2026-03-20 16:26:25.837532 | controller | - ' else' 2026-03-20 16:26:25.837536 | controller | - ' exit 1' 2026-03-20 16:26:25.837540 | controller | - ' fi' 2026-03-20 16:26:25.837545 | controller | - ' ]' 2026-03-20 16:26:25.837549 | controller | - ' logger.go:42: 16:17:39 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837553 | controller | + oc project watcher-kuttl-default' 2026-03-20 16:26:25.837558 | controller | - ' logger.go:42: 16:17:39 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837562 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.837566 | controller | - ' logger.go:42: 16:17:39 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837570 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.837575 | controller | - ' logger.go:42: 16:17:39 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837579 | controller | ++ head -1' 2026-03-20 16:26:25.837583 | controller | - ' logger.go:42: 16:17:39 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837587 | controller | ++ grep -v ''^$''' 2026-03-20 16:26:25.837591 | controller | - ' logger.go:42: 16:17:39 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837595 | controller | + APIPOD=' 2026-03-20 16:26:25.837600 | controller | - ' logger.go:42: 16:17:40 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837604 | controller | running command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.837608 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.837612 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.837617 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.837621 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.837625 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.837629 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-20 16:26:25.837634 | controller | == 1 ]' 2026-03-20 16:26:25.837638 | controller | - ' else' 2026-03-20 16:26:25.837644 | controller | - ' exit 1' 2026-03-20 16:26:25.837648 | controller | - ' fi' 2026-03-20 16:26:25.837652 | controller | - ' ]' 2026-03-20 16:26:25.837657 | controller | - ' logger.go:42: 16:17:40 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837661 | controller | + oc project watcher-kuttl-default' 2026-03-20 16:26:25.837665 | controller | - ' logger.go:42: 16:17:40 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837669 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.837673 | controller | - ' logger.go:42: 16:17:40 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837678 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.837682 | controller | - ' logger.go:42: 16:17:40 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837686 | controller | ++ head -1' 2026-03-20 16:26:25.837690 | controller | - ' logger.go:42: 16:17:40 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837710 | controller | ++ grep -v ''^$''' 2026-03-20 16:26:25.837714 | controller | - ' logger.go:42: 16:17:40 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837719 | controller | + APIPOD=' 2026-03-20 16:26:25.837723 | controller | - ' logger.go:42: 16:17:41 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837727 | controller | running command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.837731 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.837735 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.837742 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.837746 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.837750 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.837754 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-20 16:26:25.837759 | controller | == 1 ]' 2026-03-20 16:26:25.837763 | controller | - ' else' 2026-03-20 16:26:25.837767 | controller | - ' exit 1' 2026-03-20 16:26:25.837771 | controller | - ' fi' 2026-03-20 16:26:25.837776 | controller | - ' ]' 2026-03-20 16:26:25.837780 | controller | - ' logger.go:42: 16:17:41 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837784 | controller | + oc project watcher-kuttl-default' 2026-03-20 16:26:25.837788 | controller | - ' logger.go:42: 16:17:41 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837803 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.837808 | controller | - ' logger.go:42: 16:17:41 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837812 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.837816 | controller | - ' logger.go:42: 16:17:41 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837821 | controller | ++ head -1' 2026-03-20 16:26:25.837825 | controller | - ' logger.go:42: 16:17:41 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837829 | controller | ++ grep -v ''^$''' 2026-03-20 16:26:25.837833 | controller | - ' logger.go:42: 16:17:42 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837837 | controller | + APIPOD=' 2026-03-20 16:26:25.837842 | controller | - ' logger.go:42: 16:17:43 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837846 | controller | running command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.837851 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.837856 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.837862 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.837868 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.837874 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.837879 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-20 16:26:25.837884 | controller | == 1 ]' 2026-03-20 16:26:25.837888 | controller | - ' else' 2026-03-20 16:26:25.837892 | controller | - ' exit 1' 2026-03-20 16:26:25.837896 | controller | - ' fi' 2026-03-20 16:26:25.837900 | controller | - ' ]' 2026-03-20 16:26:25.837905 | controller | - ' logger.go:42: 16:17:43 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837909 | controller | + oc project watcher-kuttl-default' 2026-03-20 16:26:25.837913 | controller | - ' logger.go:42: 16:17:43 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837917 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.837922 | controller | - ' logger.go:42: 16:17:43 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837928 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.837934 | controller | - ' logger.go:42: 16:17:43 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837939 | controller | ++ head -1' 2026-03-20 16:26:25.837945 | controller | - ' logger.go:42: 16:17:43 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837951 | controller | ++ grep -v ''^$''' 2026-03-20 16:26:25.837956 | controller | - ' logger.go:42: 16:17:43 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837961 | controller | + APIPOD=' 2026-03-20 16:26:25.837967 | controller | - ' logger.go:42: 16:17:44 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.837971 | controller | running command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.837976 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.837980 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.837984 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.837988 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.837993 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.837997 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-20 16:26:25.838001 | controller | == 1 ]' 2026-03-20 16:26:25.838005 | controller | - ' else' 2026-03-20 16:26:25.838009 | controller | - ' exit 1' 2026-03-20 16:26:25.838013 | controller | - ' fi' 2026-03-20 16:26:25.838018 | controller | - ' ]' 2026-03-20 16:26:25.838022 | controller | - ' logger.go:42: 16:17:44 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838026 | controller | + oc project watcher-kuttl-default' 2026-03-20 16:26:25.838030 | controller | - ' logger.go:42: 16:17:44 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838035 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.838039 | controller | - ' logger.go:42: 16:17:44 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838043 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.838047 | controller | - ' logger.go:42: 16:17:44 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838052 | controller | ++ head -1' 2026-03-20 16:26:25.838056 | controller | - ' logger.go:42: 16:17:44 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838060 | controller | ++ grep -v ''^$''' 2026-03-20 16:26:25.838064 | controller | - ' logger.go:42: 16:17:44 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838068 | controller | + APIPOD=' 2026-03-20 16:26:25.838073 | controller | - ' logger.go:42: 16:17:45 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838079 | controller | running command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.838085 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.838090 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.838096 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.838102 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.838108 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.838123 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-20 16:26:25.838128 | controller | == 1 ]' 2026-03-20 16:26:25.838132 | controller | - ' else' 2026-03-20 16:26:25.838136 | controller | - ' exit 1' 2026-03-20 16:26:25.838140 | controller | - ' fi' 2026-03-20 16:26:25.838145 | controller | - ' ]' 2026-03-20 16:26:25.838149 | controller | - ' logger.go:42: 16:17:45 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838153 | controller | + oc project watcher-kuttl-default' 2026-03-20 16:26:25.838157 | controller | - ' logger.go:42: 16:17:45 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838162 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.838166 | controller | - ' logger.go:42: 16:17:45 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838170 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.838174 | controller | - ' logger.go:42: 16:17:45 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838178 | controller | ++ head -1' 2026-03-20 16:26:25.838186 | controller | - ' logger.go:42: 16:17:45 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838190 | controller | ++ grep -v ''^$''' 2026-03-20 16:26:25.838198 | controller | - ' logger.go:42: 16:17:46 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838203 | controller | + APIPOD=' 2026-03-20 16:26:25.838207 | controller | - ' logger.go:42: 16:17:47 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838211 | controller | running command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.838215 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.838219 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.838223 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.838228 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.838234 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.838239 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-20 16:26:25.838243 | controller | == 1 ]' 2026-03-20 16:26:25.838247 | controller | - ' else' 2026-03-20 16:26:25.838251 | controller | - ' exit 1' 2026-03-20 16:26:25.838256 | controller | - ' fi' 2026-03-20 16:26:25.838261 | controller | - ' ]' 2026-03-20 16:26:25.838267 | controller | - ' logger.go:42: 16:17:47 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838272 | controller | + oc project watcher-kuttl-default' 2026-03-20 16:26:25.838278 | controller | - ' logger.go:42: 16:17:47 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838283 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.838288 | controller | - ' logger.go:42: 16:17:47 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838292 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.838296 | controller | - ' logger.go:42: 16:17:47 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838300 | controller | ++ head -1' 2026-03-20 16:26:25.838305 | controller | - ' logger.go:42: 16:17:47 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838309 | controller | ++ grep -v ''^$''' 2026-03-20 16:26:25.838313 | controller | - ' logger.go:42: 16:17:47 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838318 | controller | + APIPOD=' 2026-03-20 16:26:25.838324 | controller | - ' logger.go:42: 16:17:48 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838330 | controller | running command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.838336 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.838341 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.838345 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.838349 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.838354 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.838358 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-20 16:26:25.838362 | controller | == 1 ]' 2026-03-20 16:26:25.838366 | controller | - ' else' 2026-03-20 16:26:25.838370 | controller | - ' exit 1' 2026-03-20 16:26:25.838375 | controller | - ' fi' 2026-03-20 16:26:25.838379 | controller | - ' ]' 2026-03-20 16:26:25.838383 | controller | - ' logger.go:42: 16:17:48 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838387 | controller | + oc project watcher-kuttl-default' 2026-03-20 16:26:25.838391 | controller | - ' logger.go:42: 16:17:48 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838395 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.838400 | controller | - ' logger.go:42: 16:17:48 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838407 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.838411 | controller | - ' logger.go:42: 16:17:48 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838415 | controller | ++ head -1' 2026-03-20 16:26:25.838419 | controller | - ' logger.go:42: 16:17:48 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838424 | controller | ++ grep -v ''^$''' 2026-03-20 16:26:25.838428 | controller | - ' logger.go:42: 16:17:48 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838432 | controller | + APIPOD=' 2026-03-20 16:26:25.838436 | controller | - ' logger.go:42: 16:17:49 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838449 | controller | running command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.838454 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.838458 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.838462 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.838466 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.838471 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.838476 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-20 16:26:25.838481 | controller | == 1 ]' 2026-03-20 16:26:25.838487 | controller | - ' else' 2026-03-20 16:26:25.838493 | controller | - ' exit 1' 2026-03-20 16:26:25.838498 | controller | - ' fi' 2026-03-20 16:26:25.838504 | controller | - ' ]' 2026-03-20 16:26:25.838510 | controller | - ' logger.go:42: 16:17:49 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838515 | controller | + oc project watcher-kuttl-default' 2026-03-20 16:26:25.838521 | controller | - ' logger.go:42: 16:17:49 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838527 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.838532 | controller | - ' logger.go:42: 16:17:49 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838538 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.838544 | controller | - ' logger.go:42: 16:17:49 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838550 | controller | ++ head -1' 2026-03-20 16:26:25.838555 | controller | - ' logger.go:42: 16:17:49 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838559 | controller | ++ grep -v ''^$''' 2026-03-20 16:26:25.838563 | controller | - ' logger.go:42: 16:17:50 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838567 | controller | + APIPOD=' 2026-03-20 16:26:25.838571 | controller | - ' logger.go:42: 16:17:51 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838576 | controller | running command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.838580 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.838584 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.838588 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.838592 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.838596 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.838601 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-20 16:26:25.838605 | controller | == 1 ]' 2026-03-20 16:26:25.838610 | controller | - ' else' 2026-03-20 16:26:25.838617 | controller | - ' exit 1' 2026-03-20 16:26:25.838623 | controller | - ' fi' 2026-03-20 16:26:25.838628 | controller | - ' ]' 2026-03-20 16:26:25.838634 | controller | - ' logger.go:42: 16:17:51 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838644 | controller | + oc project watcher-kuttl-default' 2026-03-20 16:26:25.838652 | controller | - ' logger.go:42: 16:17:51 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838658 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.838664 | controller | - ' logger.go:42: 16:17:51 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838669 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.838673 | controller | - ' logger.go:42: 16:17:51 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838677 | controller | ++ head -1' 2026-03-20 16:26:25.838681 | controller | - ' logger.go:42: 16:17:51 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838687 | controller | ++ grep -v ''^$''' 2026-03-20 16:26:25.838714 | controller | - ' logger.go:42: 16:17:51 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838726 | controller | + APIPOD=watcher-kuttl-api-0' 2026-03-20 16:26:25.838733 | controller | - ' logger.go:42: 16:17:51 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838739 | controller | + ''['' -n watcher-kuttl-api-0 '']''' 2026-03-20 16:26:25.838745 | controller | - ' logger.go:42: 16:17:51 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838750 | controller | ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+=''' 2026-03-20 16:26:25.838756 | controller | - ' logger.go:42: 16:17:51 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838762 | controller | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.838768 | controller | - ' logger.go:42: 16:17:51 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838773 | controller | error: unable to upgrade connection: container not found ("watcher-api")' 2026-03-20 16:26:25.838779 | controller | - ' logger.go:42: 16:17:51 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838784 | controller | ++ echo' 2026-03-20 16:26:25.838789 | controller | - ' logger.go:42: 16:17:51 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838793 | controller | + ''['' 0 == 1 '']''' 2026-03-20 16:26:25.838798 | controller | - ' logger.go:42: 16:17:52 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838804 | controller | running command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.838809 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.838815 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.838831 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.838837 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.838843 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.838848 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-20 16:26:25.838854 | controller | == 1 ]' 2026-03-20 16:26:25.838859 | controller | - ' else' 2026-03-20 16:26:25.838868 | controller | - ' exit 1' 2026-03-20 16:26:25.838873 | controller | - ' fi' 2026-03-20 16:26:25.838878 | controller | - ' ]' 2026-03-20 16:26:25.838882 | controller | - ' logger.go:42: 16:17:52 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838887 | controller | + oc project watcher-kuttl-default' 2026-03-20 16:26:25.838891 | controller | - ' logger.go:42: 16:17:52 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838895 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.838899 | controller | - ' logger.go:42: 16:17:52 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838903 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.838908 | controller | - ' logger.go:42: 16:17:52 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838914 | controller | ++ head -1' 2026-03-20 16:26:25.838918 | controller | - ' logger.go:42: 16:17:52 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838922 | controller | ++ grep -v ''^$''' 2026-03-20 16:26:25.838927 | controller | - ' logger.go:42: 16:17:53 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838931 | controller | + APIPOD=watcher-kuttl-api-0' 2026-03-20 16:26:25.838935 | controller | - ' logger.go:42: 16:17:53 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838939 | controller | + ''['' -n watcher-kuttl-api-0 '']''' 2026-03-20 16:26:25.838943 | controller | - ' logger.go:42: 16:17:53 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838947 | controller | ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+=''' 2026-03-20 16:26:25.838952 | controller | - ' logger.go:42: 16:17:53 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838956 | controller | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.838960 | controller | - ' logger.go:42: 16:17:53 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.838966 | controller | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.838971 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.838975 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.838980 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_a660:8502e801b18e815763860e7eefc70339@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-20 16:26:25.838984 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-20 16:26:25.838989 | controller | driver = messagingv2 transport_url = ''rabbit://default_user_I6f6zKOBLU76Pr74C5S:NMvNvliR9JfNJwTtRx2-upsPmWKjcAkC@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.838994 | controller | ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.838998 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.839002 | controller | project_domain_name = Default project_name = service user_domain_name = Default 2026-03-20 16:26:25.839007 | controller | password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 2026-03-20 16:26:25.839013 | controller | interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.839017 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-20 16:26:25.839021 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.839025 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.839029 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-20 16:26:25.839034 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-20 16:26:25.839038 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.839042 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.839046 | controller | tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc 2026-03-20 16:26:25.839052 | controller | port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' 2026-03-20 16:26:25.839067 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-20 16:26:25.839071 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-20 16:26:25.839076 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-20 16:26:25.839080 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-20 16:26:25.839084 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-20 16:26:25.839088 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-20 16:26:25.839093 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-20 16:26:25.839099 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-20 16:26:25.839103 | controller | - ' logger.go:42: 16:17:53 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.839107 | controller | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.839111 | controller | - ' logger.go:42: 16:17:54 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.839116 | controller | running command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.839120 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.839124 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.839128 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.839134 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.839139 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.839146 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-20 16:26:25.839152 | controller | == 1 ]' 2026-03-20 16:26:25.839156 | controller | - ' else' 2026-03-20 16:26:25.839160 | controller | - ' exit 1' 2026-03-20 16:26:25.839164 | controller | - ' fi' 2026-03-20 16:26:25.839169 | controller | - ' ]' 2026-03-20 16:26:25.839173 | controller | - ' logger.go:42: 16:17:54 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.839177 | controller | + oc project watcher-kuttl-default' 2026-03-20 16:26:25.839181 | controller | - ' logger.go:42: 16:17:54 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.839185 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.839190 | controller | - ' logger.go:42: 16:17:54 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.839194 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.839200 | controller | - ' logger.go:42: 16:17:54 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.839205 | controller | ++ head -1' 2026-03-20 16:26:25.839210 | controller | - ' logger.go:42: 16:17:54 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.839214 | controller | ++ grep -v ''^$''' 2026-03-20 16:26:25.839218 | controller | - ' logger.go:42: 16:17:54 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.839223 | controller | + APIPOD=watcher-kuttl-api-0' 2026-03-20 16:26:25.839228 | controller | - ' logger.go:42: 16:17:54 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.839233 | controller | + ''['' -n watcher-kuttl-api-0 '']''' 2026-03-20 16:26:25.839239 | controller | - ' logger.go:42: 16:17:54 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.839244 | controller | ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+=''' 2026-03-20 16:26:25.839250 | controller | - ' logger.go:42: 16:17:54 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.839255 | controller | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.839263 | controller | - ' logger.go:42: 16:17:54 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.839268 | controller | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.839274 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.839279 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.839284 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_a660:8502e801b18e815763860e7eefc70339@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-20 16:26:25.839290 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-20 16:26:25.839295 | controller | driver = messagingv2 transport_url = ''rabbit://default_user_I6f6zKOBLU76Pr74C5S:NMvNvliR9JfNJwTtRx2-upsPmWKjcAkC@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.839301 | controller | ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.839306 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.839320 | controller | project_domain_name = Default project_name = service user_domain_name = Default 2026-03-20 16:26:25.839326 | controller | password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 2026-03-20 16:26:25.839334 | controller | interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.839340 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-20 16:26:25.839345 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.839349 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.839354 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-20 16:26:25.839358 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-20 16:26:25.839362 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.839366 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.839370 | controller | tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc 2026-03-20 16:26:25.839375 | controller | port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' 2026-03-20 16:26:25.839379 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-20 16:26:25.839383 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-20 16:26:25.839387 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-20 16:26:25.839391 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-20 16:26:25.839396 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-20 16:26:25.839400 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-20 16:26:25.839404 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-20 16:26:25.839411 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-20 16:26:25.839416 | controller | - ' logger.go:42: 16:17:54 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.839423 | controller | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.839427 | controller | - ' logger.go:42: 16:17:55 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.839431 | controller | running command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.839437 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.839443 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.839449 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.839455 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.839460 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.839481 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-20 16:26:25.839485 | controller | == 1 ]' 2026-03-20 16:26:25.839490 | controller | - ' else' 2026-03-20 16:26:25.839494 | controller | - ' exit 1' 2026-03-20 16:26:25.839498 | controller | - ' fi' 2026-03-20 16:26:25.839502 | controller | - ' ]' 2026-03-20 16:26:25.839507 | controller | - ' logger.go:42: 16:17:55 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.839511 | controller | + oc project watcher-kuttl-default' 2026-03-20 16:26:25.839515 | controller | - ' logger.go:42: 16:17:56 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.839519 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.839523 | controller | - ' logger.go:42: 16:17:56 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.839528 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.839532 | controller | - ' logger.go:42: 16:17:56 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.839536 | controller | ++ head -1' 2026-03-20 16:26:25.839540 | controller | - ' logger.go:42: 16:17:56 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.839544 | controller | ++ grep -v ''^$''' 2026-03-20 16:26:25.839549 | controller | - ' logger.go:42: 16:17:56 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.839553 | controller | + APIPOD=watcher-kuttl-api-0' 2026-03-20 16:26:25.839557 | controller | - ' logger.go:42: 16:17:56 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.839561 | controller | + ''['' -n watcher-kuttl-api-0 '']''' 2026-03-20 16:26:25.839565 | controller | - ' logger.go:42: 16:17:56 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.839570 | controller | ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+=''' 2026-03-20 16:26:25.839574 | controller | - ' logger.go:42: 16:17:56 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.839586 | controller | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.839591 | controller | - ' logger.go:42: 16:17:56 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.839595 | controller | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.839599 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.839603 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.839608 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_a660:8502e801b18e815763860e7eefc70339@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-20 16:26:25.839612 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-20 16:26:25.839616 | controller | driver = messagingv2 transport_url = ''rabbit://default_user_I6f6zKOBLU76Pr74C5S:NMvNvliR9JfNJwTtRx2-upsPmWKjcAkC@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.839623 | controller | ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.839627 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.839631 | controller | project_domain_name = Default project_name = service user_domain_name = Default 2026-03-20 16:26:25.839635 | controller | password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 2026-03-20 16:26:25.839640 | controller | interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.839644 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-20 16:26:25.839648 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.839652 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.839656 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-20 16:26:25.839660 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-20 16:26:25.839665 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.839669 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.839673 | controller | tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc 2026-03-20 16:26:25.839677 | controller | port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' 2026-03-20 16:26:25.839681 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-20 16:26:25.839686 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-20 16:26:25.839690 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-20 16:26:25.839710 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-20 16:26:25.839714 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-20 16:26:25.839718 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-20 16:26:25.839723 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-20 16:26:25.839727 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-20 16:26:25.839731 | controller | - ' logger.go:42: 16:17:56 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.839735 | controller | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.839740 | controller | - ' logger.go:42: 16:17:57 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.839744 | controller | running command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.839748 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.839770 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.839777 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.839783 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.839787 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.839792 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-20 16:26:25.839796 | controller | == 1 ]' 2026-03-20 16:26:25.839800 | controller | - ' else' 2026-03-20 16:26:25.839804 | controller | - ' exit 1' 2026-03-20 16:26:25.839808 | controller | - ' fi' 2026-03-20 16:26:25.839815 | controller | - ' ]' 2026-03-20 16:26:25.839819 | controller | - ' logger.go:42: 16:17:57 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.839824 | controller | + oc project watcher-kuttl-default' 2026-03-20 16:26:25.839837 | controller | - ' logger.go:42: 16:17:57 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.839841 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.839845 | controller | - ' logger.go:42: 16:17:57 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.839850 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.839854 | controller | - ' logger.go:42: 16:17:57 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.839858 | controller | ++ head -1' 2026-03-20 16:26:25.839862 | controller | - ' logger.go:42: 16:17:57 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.839866 | controller | ++ grep -v ''^$''' 2026-03-20 16:26:25.839871 | controller | - ' logger.go:42: 16:17:57 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.839875 | controller | + APIPOD=watcher-kuttl-api-0' 2026-03-20 16:26:25.839879 | controller | - ' logger.go:42: 16:17:57 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.839883 | controller | + ''['' -n watcher-kuttl-api-0 '']''' 2026-03-20 16:26:25.839887 | controller | - ' logger.go:42: 16:17:57 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.839894 | controller | ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+=''' 2026-03-20 16:26:25.839899 | controller | - ' logger.go:42: 16:17:57 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.839904 | controller | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.839908 | controller | - ' logger.go:42: 16:17:57 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.839912 | controller | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.839916 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.839920 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.839925 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_a660:8502e801b18e815763860e7eefc70339@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-20 16:26:25.839929 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-20 16:26:25.839933 | controller | driver = messagingv2 transport_url = ''rabbit://default_user_I6f6zKOBLU76Pr74C5S:NMvNvliR9JfNJwTtRx2-upsPmWKjcAkC@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.839937 | controller | ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.839941 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.839946 | controller | project_domain_name = Default project_name = service user_domain_name = Default 2026-03-20 16:26:25.839950 | controller | password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 2026-03-20 16:26:25.839957 | controller | interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.839961 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-20 16:26:25.839965 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.839969 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.839975 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-20 16:26:25.839979 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-20 16:26:25.839984 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.839988 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.839992 | controller | tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc 2026-03-20 16:26:25.839996 | controller | port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' 2026-03-20 16:26:25.840000 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-20 16:26:25.840004 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-20 16:26:25.840009 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-20 16:26:25.840015 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-20 16:26:25.840019 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-20 16:26:25.840031 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-20 16:26:25.840035 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-20 16:26:25.840040 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-20 16:26:25.840044 | controller | - ' logger.go:42: 16:17:57 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.840048 | controller | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.840052 | controller | - ' logger.go:42: 16:17:59 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.840056 | controller | running command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.840061 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.840065 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.840069 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.840073 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.840077 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.840081 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-20 16:26:25.840086 | controller | == 1 ]' 2026-03-20 16:26:25.840090 | controller | - ' else' 2026-03-20 16:26:25.840094 | controller | - ' exit 1' 2026-03-20 16:26:25.840100 | controller | - ' fi' 2026-03-20 16:26:25.840106 | controller | - ' ]' 2026-03-20 16:26:25.840112 | controller | - ' logger.go:42: 16:17:59 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.840118 | controller | + oc project watcher-kuttl-default' 2026-03-20 16:26:25.840123 | controller | - ' logger.go:42: 16:17:59 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.840128 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.840132 | controller | - ' logger.go:42: 16:17:59 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.840136 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.840140 | controller | - ' logger.go:42: 16:17:59 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.840144 | controller | ++ head -1' 2026-03-20 16:26:25.840149 | controller | - ' logger.go:42: 16:17:59 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.840153 | controller | ++ grep -v ''^$''' 2026-03-20 16:26:25.840157 | controller | - ' logger.go:42: 16:17:59 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.840163 | controller | + APIPOD=watcher-kuttl-api-0' 2026-03-20 16:26:25.840167 | controller | - ' logger.go:42: 16:17:59 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.840172 | controller | + ''['' -n watcher-kuttl-api-0 '']''' 2026-03-20 16:26:25.840176 | controller | - ' logger.go:42: 16:17:59 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.840181 | controller | ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+=''' 2026-03-20 16:26:25.840186 | controller | - ' logger.go:42: 16:17:59 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.840192 | controller | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.840198 | controller | - ' logger.go:42: 16:17:59 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.840203 | controller | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.840209 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.840215 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.840221 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_a660:8502e801b18e815763860e7eefc70339@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-20 16:26:25.840227 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-20 16:26:25.840233 | controller | driver = messagingv2 transport_url = ''rabbit://default_user_I6f6zKOBLU76Pr74C5S:NMvNvliR9JfNJwTtRx2-upsPmWKjcAkC@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.840239 | controller | ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.840244 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.840250 | controller | project_domain_name = Default project_name = service user_domain_name = Default 2026-03-20 16:26:25.840256 | controller | password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 2026-03-20 16:26:25.840265 | controller | interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.840269 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-20 16:26:25.840274 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.840291 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.840298 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-20 16:26:25.840303 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-20 16:26:25.840309 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.840315 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.840319 | controller | tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc 2026-03-20 16:26:25.840324 | controller | port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' 2026-03-20 16:26:25.840328 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-20 16:26:25.840332 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-20 16:26:25.840336 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-20 16:26:25.840343 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-20 16:26:25.840348 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-20 16:26:25.840352 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-20 16:26:25.840356 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-20 16:26:25.840360 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-20 16:26:25.840365 | controller | - ' logger.go:42: 16:17:59 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.840369 | controller | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.840373 | controller | - ' logger.go:42: 16:18:00 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.840377 | controller | running command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.840381 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.840386 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.840390 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.840394 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.840398 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.840403 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-20 16:26:25.840407 | controller | == 1 ]' 2026-03-20 16:26:25.840411 | controller | - ' else' 2026-03-20 16:26:25.840415 | controller | - ' exit 1' 2026-03-20 16:26:25.840419 | controller | - ' fi' 2026-03-20 16:26:25.840424 | controller | - ' ]' 2026-03-20 16:26:25.840428 | controller | - ' logger.go:42: 16:18:00 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.840432 | controller | + oc project watcher-kuttl-default' 2026-03-20 16:26:25.840437 | controller | - ' logger.go:42: 16:18:00 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.840441 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.840445 | controller | - ' logger.go:42: 16:18:00 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.840449 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.840453 | controller | - ' logger.go:42: 16:18:00 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.840458 | controller | ++ head -1' 2026-03-20 16:26:25.840462 | controller | - ' logger.go:42: 16:18:00 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.840466 | controller | ++ grep -v ''^$''' 2026-03-20 16:26:25.840470 | controller | - ' logger.go:42: 16:18:00 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.840474 | controller | + APIPOD=watcher-kuttl-api-0' 2026-03-20 16:26:25.840479 | controller | - ' logger.go:42: 16:18:00 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.840483 | controller | + ''['' -n watcher-kuttl-api-0 '']''' 2026-03-20 16:26:25.840487 | controller | - ' logger.go:42: 16:18:00 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.840491 | controller | ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+=''' 2026-03-20 16:26:25.840496 | controller | - ' logger.go:42: 16:18:00 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.840500 | controller | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.840504 | controller | - ' logger.go:42: 16:18:01 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.840508 | controller | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.840524 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.840529 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.840544 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_a660:8502e801b18e815763860e7eefc70339@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-20 16:26:25.840550 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-20 16:26:25.840554 | controller | driver = messagingv2 transport_url = ''rabbit://default_user_I6f6zKOBLU76Pr74C5S:NMvNvliR9JfNJwTtRx2-upsPmWKjcAkC@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.840559 | controller | ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.840563 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.840567 | controller | project_domain_name = Default project_name = service user_domain_name = Default 2026-03-20 16:26:25.840571 | controller | password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 2026-03-20 16:26:25.840576 | controller | interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.840580 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-20 16:26:25.840586 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.840591 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.840597 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-20 16:26:25.840603 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-20 16:26:25.840608 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.840614 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.840621 | controller | tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc 2026-03-20 16:26:25.840626 | controller | port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' 2026-03-20 16:26:25.840632 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-20 16:26:25.840638 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-20 16:26:25.840644 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-20 16:26:25.840650 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-20 16:26:25.840656 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-20 16:26:25.840663 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-20 16:26:25.840669 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-20 16:26:25.840675 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-20 16:26:25.840681 | controller | - ' logger.go:42: 16:18:01 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.840687 | controller | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.840717 | controller | - ' logger.go:42: 16:18:02 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.840729 | controller | running command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.840742 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.840749 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.840755 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.840760 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.840766 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.840772 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-20 16:26:25.840778 | controller | == 1 ]' 2026-03-20 16:26:25.840782 | controller | - ' else' 2026-03-20 16:26:25.840786 | controller | - ' exit 1' 2026-03-20 16:26:25.840790 | controller | - ' fi' 2026-03-20 16:26:25.840795 | controller | - ' ]' 2026-03-20 16:26:25.840801 | controller | - ' logger.go:42: 16:18:02 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.840807 | controller | + oc project watcher-kuttl-default' 2026-03-20 16:26:25.840813 | controller | - ' logger.go:42: 16:18:02 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.840819 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.840825 | controller | - ' logger.go:42: 16:18:02 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.840830 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.840836 | controller | - ' logger.go:42: 16:18:02 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.840842 | controller | ++ head -1' 2026-03-20 16:26:25.840861 | controller | - ' logger.go:42: 16:18:02 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.840868 | controller | ++ grep -v ''^$''' 2026-03-20 16:26:25.840874 | controller | - ' logger.go:42: 16:18:02 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.840880 | controller | + APIPOD=watcher-kuttl-api-0' 2026-03-20 16:26:25.840886 | controller | - ' logger.go:42: 16:18:02 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.840892 | controller | + ''['' -n watcher-kuttl-api-0 '']''' 2026-03-20 16:26:25.840898 | controller | - ' logger.go:42: 16:18:02 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.840904 | controller | ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+=''' 2026-03-20 16:26:25.840910 | controller | - ' logger.go:42: 16:18:02 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.840915 | controller | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.840919 | controller | - ' logger.go:42: 16:18:02 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.840924 | controller | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.840928 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.840932 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.840936 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_a660:8502e801b18e815763860e7eefc70339@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-20 16:26:25.840941 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-20 16:26:25.840945 | controller | driver = messagingv2 transport_url = ''rabbit://default_user_I6f6zKOBLU76Pr74C5S:NMvNvliR9JfNJwTtRx2-upsPmWKjcAkC@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.840949 | controller | ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.840953 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.840961 | controller | project_domain_name = Default project_name = service user_domain_name = Default 2026-03-20 16:26:25.840965 | controller | password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 2026-03-20 16:26:25.840969 | controller | interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.840974 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-20 16:26:25.840978 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.840982 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.840989 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-20 16:26:25.840994 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-20 16:26:25.840998 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.841002 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.841006 | controller | tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc 2026-03-20 16:26:25.841012 | controller | port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' 2026-03-20 16:26:25.841016 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-20 16:26:25.841021 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-20 16:26:25.841025 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-20 16:26:25.841029 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-20 16:26:25.841033 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-20 16:26:25.841038 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-20 16:26:25.841042 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-20 16:26:25.841046 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-20 16:26:25.841050 | controller | - ' logger.go:42: 16:18:02 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.841055 | controller | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.841061 | controller | - ' logger.go:42: 16:18:03 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.841076 | controller | running command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.841083 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.841089 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.841095 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.841101 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.841107 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.841112 | controller | |grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+='') 2026-03-20 16:26:25.841118 | controller | == 1 ]' 2026-03-20 16:26:25.841124 | controller | - ' else' 2026-03-20 16:26:25.841128 | controller | - ' exit 1' 2026-03-20 16:26:25.841133 | controller | - ' fi' 2026-03-20 16:26:25.841137 | controller | - ' ]' 2026-03-20 16:26:25.841141 | controller | - ' logger.go:42: 16:18:03 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.841145 | controller | + oc project watcher-kuttl-default' 2026-03-20 16:26:25.841150 | controller | - ' logger.go:42: 16:18:03 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.841156 | controller | Already on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.841161 | controller | - ' logger.go:42: 16:18:03 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.841165 | controller | ++ oc get pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.841169 | controller | - ' logger.go:42: 16:18:03 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.841173 | controller | ++ head -1' 2026-03-20 16:26:25.841178 | controller | - ' logger.go:42: 16:18:03 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.841182 | controller | ++ grep -v ''^$''' 2026-03-20 16:26:25.841186 | controller | - ' logger.go:42: 16:18:04 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.841190 | controller | + APIPOD=watcher-kuttl-api-0' 2026-03-20 16:26:25.841194 | controller | - ' logger.go:42: 16:18:04 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.841199 | controller | + ''['' -n watcher-kuttl-api-0 '']''' 2026-03-20 16:26:25.841203 | controller | - ' logger.go:42: 16:18:04 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.841207 | controller | ++ grep -czPo ''\[oslo_messaging_notifications\]\s+driver\s+=\s+messagingv2\s+transport_url\s+=''' 2026-03-20 16:26:25.841211 | controller | - ' logger.go:42: 16:18:04 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.841215 | controller | +++ oc rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.841219 | controller | - ' logger.go:42: 16:18:04 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.841224 | controller | ++ echo ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.841228 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.841232 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.841236 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_a660:8502e801b18e815763860e7eefc70339@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-20 16:26:25.841243 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-20 16:26:25.841247 | controller | driver = messagingv2 transport_url = ''rabbit://default_user_I6f6zKOBLU76Pr74C5S:NMvNvliR9JfNJwTtRx2-upsPmWKjcAkC@rabbitmq-notifications.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.841252 | controller | ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.841256 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.841260 | controller | project_domain_name = Default project_name = service user_domain_name = Default 2026-03-20 16:26:25.841264 | controller | password = password username = watcher auth_type = password auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 2026-03-20 16:26:25.841270 | controller | interface = internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.841275 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-20 16:26:25.841279 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.841283 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.841287 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-20 16:26:25.841292 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-20 16:26:25.841296 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.841303 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.841315 | controller | tls_enabled=true memcache_dead_retry = 30 ''[prometheus_client]'' host = metric-storage-prometheus.watcher-kuttl-default.svc 2026-03-20 16:26:25.841320 | controller | port = 9090 cafile = /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' 2026-03-20 16:26:25.841324 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-20 16:26:25.841329 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-20 16:26:25.841333 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-20 16:26:25.841337 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-20 16:26:25.841342 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-20 16:26:25.841346 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-20 16:26:25.841352 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-20 16:26:25.841361 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-20 16:26:25.841367 | controller | - ' logger.go:42: 16:18:04 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.841372 | controller | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.841378 | controller | - ' logger.go:42: 16:18:04 | watcher-notification/1-deploy-with-notification | 2026-03-20 16:26:25.841384 | controller | test step completed 1-deploy-with-notification' 2026-03-20 16:26:25.841390 | controller | - ' logger.go:42: 16:18:04 | watcher-notification/2-cleanup-watcher | starting 2026-03-20 16:26:25.841396 | controller | test step 2-cleanup-watcher' 2026-03-20 16:26:25.841402 | controller | - ' logger.go:42: 16:18:11 | watcher-notification/2-cleanup-watcher | test step 2026-03-20 16:26:25.841408 | controller | completed 2-cleanup-watcher' 2026-03-20 16:26:25.841414 | controller | - ' logger.go:42: 16:18:11 | watcher-notification | skipping kubernetes event logging' 2026-03-20 16:26:25.841420 | controller | - === CONT kuttl/harness/watcher-topology 2026-03-20 16:26:25.841426 | controller | - ' logger.go:42: 16:18:11 | watcher-topology | Skipping creation of user-supplied 2026-03-20 16:26:25.841432 | controller | namespace: watcher-kuttl-default' 2026-03-20 16:26:25.841438 | controller | - ' logger.go:42: 16:18:11 | watcher-topology/0-cleanup-watcher | starting test 2026-03-20 16:26:25.841443 | controller | step 0-cleanup-watcher' 2026-03-20 16:26:25.841449 | controller | - ' logger.go:42: 16:18:11 | watcher-topology/0-cleanup-watcher | test step completed 2026-03-20 16:26:25.841455 | controller | 0-cleanup-watcher' 2026-03-20 16:26:25.841461 | controller | - ' logger.go:42: 16:18:11 | watcher-topology/1-deploy-with-topology | starting 2026-03-20 16:26:25.841467 | controller | test step 1-deploy-with-topology' 2026-03-20 16:26:25.841474 | controller | - ' logger.go:42: 16:18:11 | watcher-topology/1-deploy-with-topology | Topology:watcher-kuttl-default/watcher-api 2026-03-20 16:26:25.841480 | controller | created' 2026-03-20 16:26:25.841486 | controller | - ' logger.go:42: 16:18:11 | watcher-topology/1-deploy-with-topology | Watcher:watcher-kuttl-default/watcher-kuttl 2026-03-20 16:26:25.841492 | controller | created' 2026-03-20 16:26:25.841498 | controller | - ' logger.go:42: 16:18:36 | watcher-topology/1-deploy-with-topology | test step 2026-03-20 16:26:25.841504 | controller | completed 1-deploy-with-topology' 2026-03-20 16:26:25.841510 | controller | - ' logger.go:42: 16:18:36 | watcher-topology/2-cleanup-watcher | starting test 2026-03-20 16:26:25.841514 | controller | step 2-cleanup-watcher' 2026-03-20 16:26:25.841519 | controller | - ' logger.go:42: 16:18:36 | watcher-topology/2-cleanup-watcher | running command: 2026-03-20 16:26:25.841523 | controller | [sh -c set -ex' 2026-03-20 16:26:25.841530 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.841534 | controller | -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]' 2026-03-20 16:26:25.841538 | controller | - ' ]' 2026-03-20 16:26:25.841543 | controller | - ' logger.go:42: 16:18:36 | watcher-topology/2-cleanup-watcher | + oc exec -n 2026-03-20 16:26:25.841547 | controller | watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-20 16:26:25.841551 | controller | -c Type' 2026-03-20 16:26:25.841555 | controller | - ' logger.go:42: 16:18:36 | watcher-topology/2-cleanup-watcher | ++ grep -c ''^watcher''' 2026-03-20 16:26:25.841559 | controller | - ' logger.go:42: 16:18:39 | watcher-topology/2-cleanup-watcher | + ''['' 1 == 2026-03-20 16:26:25.841564 | controller | 0 '']''' 2026-03-20 16:26:25.841568 | controller | - ' logger.go:42: 16:18:40 | watcher-topology/2-cleanup-watcher | running command: 2026-03-20 16:26:25.841572 | controller | [sh -c set -ex' 2026-03-20 16:26:25.841576 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.841580 | controller | -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]' 2026-03-20 16:26:25.841585 | controller | - ' ]' 2026-03-20 16:26:25.841590 | controller | - ' logger.go:42: 16:18:40 | watcher-topology/2-cleanup-watcher | + oc exec -n 2026-03-20 16:26:25.841596 | controller | watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-20 16:26:25.841601 | controller | -c Type' 2026-03-20 16:26:25.841607 | controller | - ' logger.go:42: 16:18:40 | watcher-topology/2-cleanup-watcher | ++ grep -c ''^watcher''' 2026-03-20 16:26:25.841613 | controller | - ' logger.go:42: 16:18:42 | watcher-topology/2-cleanup-watcher | + ''['' 0 == 2026-03-20 16:26:25.841619 | controller | 0 '']''' 2026-03-20 16:26:25.841624 | controller | - ' logger.go:42: 16:18:42 | watcher-topology/2-cleanup-watcher | test step completed 2026-03-20 16:26:25.841629 | controller | 2-cleanup-watcher' 2026-03-20 16:26:25.841644 | controller | - ' logger.go:42: 16:18:42 | watcher-topology | skipping kubernetes event logging' 2026-03-20 16:26:25.841649 | controller | - === CONT kuttl/harness/watcher-tls-certs-change 2026-03-20 16:26:25.841653 | controller | - ' logger.go:42: 16:18:42 | watcher-tls-certs-change | Skipping creation of user-supplied 2026-03-20 16:26:25.841657 | controller | namespace: watcher-kuttl-default' 2026-03-20 16:26:25.841662 | controller | - ' logger.go:42: 16:18:42 | watcher-tls-certs-change/0-cleanup-watcher | starting 2026-03-20 16:26:25.841666 | controller | test step 0-cleanup-watcher' 2026-03-20 16:26:25.841670 | controller | - ' logger.go:42: 16:18:42 | watcher-tls-certs-change/0-cleanup-watcher | test 2026-03-20 16:26:25.841674 | controller | step completed 0-cleanup-watcher' 2026-03-20 16:26:25.841679 | controller | - ' logger.go:42: 16:18:42 | watcher-tls-certs-change/1-deploy-with-tlse | starting 2026-03-20 16:26:25.841683 | controller | test step 1-deploy-with-tlse' 2026-03-20 16:26:25.841687 | controller | - ' logger.go:42: 16:18:42 | watcher-tls-certs-change/1-deploy-with-tlse | Certificate:watcher-kuttl-default/watcher-internal-svc 2026-03-20 16:26:25.841712 | controller | created' 2026-03-20 16:26:25.841717 | controller | - ' logger.go:42: 16:18:42 | watcher-tls-certs-change/1-deploy-with-tlse | Certificate:watcher-kuttl-default/watcher-public-svc 2026-03-20 16:26:25.841722 | controller | created' 2026-03-20 16:26:25.841726 | controller | - ' logger.go:42: 16:18:42 | watcher-tls-certs-change/1-deploy-with-tlse | Watcher:watcher-kuttl-default/watcher-kuttl 2026-03-20 16:26:25.841731 | controller | created' 2026-03-20 16:26:25.841737 | controller | - ' logger.go:42: 16:18:43 | watcher-tls-certs-change/1-deploy-with-tlse | running 2026-03-20 16:26:25.841742 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.841747 | controller | - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-20 16:26:25.841753 | controller | cat /etc/pki/tls/certs/public.crt)' 2026-03-20 16:26:25.841758 | controller | - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-20 16:26:25.841764 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-20 16:26:25.841772 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.841778 | controller | - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' 2026-03-20 16:26:25.841783 | controller | - ' exit 1' 2026-03-20 16:26:25.841789 | controller | - ' fi' 2026-03-20 16:26:25.841794 | controller | - ' ' 2026-03-20 16:26:25.841800 | controller | - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-20 16:26:25.841805 | controller | cat /etc/pki/tls/certs/internal.crt)' 2026-03-20 16:26:25.841811 | controller | - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-20 16:26:25.841816 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-20 16:26:25.841821 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.841826 | controller | - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' 2026-03-20 16:26:25.841832 | controller | - ' exit 1' 2026-03-20 16:26:25.841837 | controller | - ' fi' 2026-03-20 16:26:25.841841 | controller | - ' ]' 2026-03-20 16:26:25.841846 | controller | - ' logger.go:42: 16:18:43 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-20 16:26:25.841850 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-20 16:26:25.841854 | controller | - ' logger.go:42: 16:18:43 | watcher-tls-certs-change/1-deploy-with-tlse | Error 2026-03-20 16:26:25.841858 | controller | from server (NotFound): pods "watcher-kuttl-api-0" not found' 2026-03-20 16:26:25.841863 | controller | - ' logger.go:42: 16:18:43 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=' 2026-03-20 16:26:25.841867 | controller | - ' logger.go:42: 16:18:44 | watcher-tls-certs-change/1-deploy-with-tlse | running 2026-03-20 16:26:25.841871 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.841875 | controller | - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-20 16:26:25.841880 | controller | cat /etc/pki/tls/certs/public.crt)' 2026-03-20 16:26:25.841884 | controller | - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-20 16:26:25.841888 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-20 16:26:25.841892 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.841897 | controller | - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' 2026-03-20 16:26:25.841901 | controller | - ' exit 1' 2026-03-20 16:26:25.841905 | controller | - ' fi' 2026-03-20 16:26:25.841909 | controller | - ' ' 2026-03-20 16:26:25.841914 | controller | - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-20 16:26:25.841918 | controller | cat /etc/pki/tls/certs/internal.crt)' 2026-03-20 16:26:25.841922 | controller | - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-20 16:26:25.841926 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-20 16:26:25.841930 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.841935 | controller | - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' 2026-03-20 16:26:25.841939 | controller | - ' exit 1' 2026-03-20 16:26:25.841943 | controller | - ' fi' 2026-03-20 16:26:25.841947 | controller | - ' ]' 2026-03-20 16:26:25.841951 | controller | - ' logger.go:42: 16:18:44 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-20 16:26:25.841956 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-20 16:26:25.841962 | controller | - ' logger.go:42: 16:18:44 | watcher-tls-certs-change/1-deploy-with-tlse | Error 2026-03-20 16:26:25.841978 | controller | from server (NotFound): pods "watcher-kuttl-api-0" not found' 2026-03-20 16:26:25.841984 | controller | - ' logger.go:42: 16:18:44 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=' 2026-03-20 16:26:25.841988 | controller | - ' logger.go:42: 16:18:45 | watcher-tls-certs-change/1-deploy-with-tlse | running 2026-03-20 16:26:25.841995 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.841999 | controller | - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-20 16:26:25.842003 | controller | cat /etc/pki/tls/certs/public.crt)' 2026-03-20 16:26:25.842008 | controller | - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-20 16:26:25.842012 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-20 16:26:25.842016 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.842020 | controller | - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' 2026-03-20 16:26:25.842024 | controller | - ' exit 1' 2026-03-20 16:26:25.842029 | controller | - ' fi' 2026-03-20 16:26:25.842033 | controller | - ' ' 2026-03-20 16:26:25.842037 | controller | - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-20 16:26:25.842041 | controller | cat /etc/pki/tls/certs/internal.crt)' 2026-03-20 16:26:25.842045 | controller | - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-20 16:26:25.842050 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-20 16:26:25.842054 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.842058 | controller | - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' 2026-03-20 16:26:25.842062 | controller | - ' exit 1' 2026-03-20 16:26:25.842066 | controller | - ' fi' 2026-03-20 16:26:25.842071 | controller | - ' ]' 2026-03-20 16:26:25.842076 | controller | - ' logger.go:42: 16:18:45 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-20 16:26:25.842082 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-20 16:26:25.842087 | controller | - ' logger.go:42: 16:18:45 | watcher-tls-certs-change/1-deploy-with-tlse | Error 2026-03-20 16:26:25.842093 | controller | from server (NotFound): pods "watcher-kuttl-api-0" not found' 2026-03-20 16:26:25.842098 | controller | - ' logger.go:42: 16:18:45 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=' 2026-03-20 16:26:25.842104 | controller | - ' logger.go:42: 16:18:46 | watcher-tls-certs-change/1-deploy-with-tlse | running 2026-03-20 16:26:25.842109 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.842114 | controller | - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-20 16:26:25.842118 | controller | cat /etc/pki/tls/certs/public.crt)' 2026-03-20 16:26:25.842122 | controller | - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-20 16:26:25.842126 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-20 16:26:25.842130 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.842138 | controller | - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' 2026-03-20 16:26:25.842142 | controller | - ' exit 1' 2026-03-20 16:26:25.842147 | controller | - ' fi' 2026-03-20 16:26:25.842151 | controller | - ' ' 2026-03-20 16:26:25.842155 | controller | - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-20 16:26:25.842159 | controller | cat /etc/pki/tls/certs/internal.crt)' 2026-03-20 16:26:25.842164 | controller | - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-20 16:26:25.842168 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-20 16:26:25.842172 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.842176 | controller | - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' 2026-03-20 16:26:25.842180 | controller | - ' exit 1' 2026-03-20 16:26:25.842185 | controller | - ' fi' 2026-03-20 16:26:25.842190 | controller | - ' ]' 2026-03-20 16:26:25.842195 | controller | - ' logger.go:42: 16:18:46 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-20 16:26:25.842200 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-20 16:26:25.842209 | controller | - ' logger.go:42: 16:18:46 | watcher-tls-certs-change/1-deploy-with-tlse | Error 2026-03-20 16:26:25.842214 | controller | from server (NotFound): pods "watcher-kuttl-api-0" not found' 2026-03-20 16:26:25.842220 | controller | - ' logger.go:42: 16:18:46 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=' 2026-03-20 16:26:25.842226 | controller | - ' logger.go:42: 16:18:48 | watcher-tls-certs-change/1-deploy-with-tlse | running 2026-03-20 16:26:25.842231 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.842241 | controller | - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-20 16:26:25.842248 | controller | cat /etc/pki/tls/certs/public.crt)' 2026-03-20 16:26:25.842254 | controller | - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-20 16:26:25.842258 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-20 16:26:25.842263 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.842269 | controller | - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' 2026-03-20 16:26:25.842275 | controller | - ' exit 1' 2026-03-20 16:26:25.842281 | controller | - ' fi' 2026-03-20 16:26:25.842287 | controller | - ' ' 2026-03-20 16:26:25.842292 | controller | - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-20 16:26:25.842298 | controller | cat /etc/pki/tls/certs/internal.crt)' 2026-03-20 16:26:25.842304 | controller | - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-20 16:26:25.842321 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-20 16:26:25.842327 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.842333 | controller | - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' 2026-03-20 16:26:25.842339 | controller | - ' exit 1' 2026-03-20 16:26:25.842345 | controller | - ' fi' 2026-03-20 16:26:25.842350 | controller | - ' ]' 2026-03-20 16:26:25.842356 | controller | - ' logger.go:42: 16:18:48 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-20 16:26:25.842362 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-20 16:26:25.842367 | controller | - ' logger.go:42: 16:18:48 | watcher-tls-certs-change/1-deploy-with-tlse | Error 2026-03-20 16:26:25.842373 | controller | from server (NotFound): pods "watcher-kuttl-api-0" not found' 2026-03-20 16:26:25.842379 | controller | - ' logger.go:42: 16:18:48 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=' 2026-03-20 16:26:25.842385 | controller | - ' logger.go:42: 16:18:49 | watcher-tls-certs-change/1-deploy-with-tlse | running 2026-03-20 16:26:25.842391 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.842397 | controller | - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-20 16:26:25.842403 | controller | cat /etc/pki/tls/certs/public.crt)' 2026-03-20 16:26:25.842407 | controller | - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-20 16:26:25.842412 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-20 16:26:25.842416 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.842420 | controller | - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' 2026-03-20 16:26:25.842424 | controller | - ' exit 1' 2026-03-20 16:26:25.842429 | controller | - ' fi' 2026-03-20 16:26:25.842434 | controller | - ' ' 2026-03-20 16:26:25.842440 | controller | - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-20 16:26:25.842446 | controller | cat /etc/pki/tls/certs/internal.crt)' 2026-03-20 16:26:25.842452 | controller | - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-20 16:26:25.842457 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-20 16:26:25.842461 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.842469 | controller | - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' 2026-03-20 16:26:25.842474 | controller | - ' exit 1' 2026-03-20 16:26:25.842478 | controller | - ' fi' 2026-03-20 16:26:25.842482 | controller | - ' ]' 2026-03-20 16:26:25.842486 | controller | - ' logger.go:42: 16:18:49 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-20 16:26:25.842490 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-20 16:26:25.842495 | controller | - ' logger.go:42: 16:18:50 | watcher-tls-certs-change/1-deploy-with-tlse | Error 2026-03-20 16:26:25.842499 | controller | from server (NotFound): pods "watcher-kuttl-api-0" not found' 2026-03-20 16:26:25.842504 | controller | - ' logger.go:42: 16:18:50 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=' 2026-03-20 16:26:25.842510 | controller | - ' logger.go:42: 16:18:51 | watcher-tls-certs-change/1-deploy-with-tlse | running 2026-03-20 16:26:25.842515 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.842521 | controller | - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-20 16:26:25.842527 | controller | cat /etc/pki/tls/certs/public.crt)' 2026-03-20 16:26:25.842532 | controller | - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-20 16:26:25.842538 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-20 16:26:25.842544 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.842549 | controller | - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' 2026-03-20 16:26:25.842555 | controller | - ' exit 1' 2026-03-20 16:26:25.842561 | controller | - ' fi' 2026-03-20 16:26:25.842565 | controller | - ' ' 2026-03-20 16:26:25.842569 | controller | - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-20 16:26:25.842573 | controller | cat /etc/pki/tls/certs/internal.crt)' 2026-03-20 16:26:25.842578 | controller | - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-20 16:26:25.842582 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-20 16:26:25.842586 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.842590 | controller | - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' 2026-03-20 16:26:25.842594 | controller | - ' exit 1' 2026-03-20 16:26:25.842599 | controller | - ' fi' 2026-03-20 16:26:25.842603 | controller | - ' ]' 2026-03-20 16:26:25.842607 | controller | - ' logger.go:42: 16:18:51 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-20 16:26:25.842611 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-20 16:26:25.842615 | controller | - ' logger.go:42: 16:18:51 | watcher-tls-certs-change/1-deploy-with-tlse | Error 2026-03-20 16:26:25.842620 | controller | from server (NotFound): pods "watcher-kuttl-api-0" not found' 2026-03-20 16:26:25.842625 | controller | - ' logger.go:42: 16:18:51 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=' 2026-03-20 16:26:25.842631 | controller | - ' logger.go:42: 16:18:52 | watcher-tls-certs-change/1-deploy-with-tlse | running 2026-03-20 16:26:25.842637 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.842642 | controller | - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-20 16:26:25.842648 | controller | cat /etc/pki/tls/certs/public.crt)' 2026-03-20 16:26:25.842664 | controller | - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-20 16:26:25.842671 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-20 16:26:25.842675 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.842679 | controller | - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' 2026-03-20 16:26:25.842683 | controller | - ' exit 1' 2026-03-20 16:26:25.842688 | controller | - ' fi' 2026-03-20 16:26:25.842712 | controller | - ' ' 2026-03-20 16:26:25.842719 | controller | - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-20 16:26:25.842724 | controller | cat /etc/pki/tls/certs/internal.crt)' 2026-03-20 16:26:25.842728 | controller | - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-20 16:26:25.842732 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-20 16:26:25.842737 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.842741 | controller | - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' 2026-03-20 16:26:25.842745 | controller | - ' exit 1' 2026-03-20 16:26:25.842752 | controller | - ' fi' 2026-03-20 16:26:25.842756 | controller | - ' ]' 2026-03-20 16:26:25.842761 | controller | - ' logger.go:42: 16:18:52 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-20 16:26:25.842765 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-20 16:26:25.842769 | controller | - ' logger.go:42: 16:18:52 | watcher-tls-certs-change/1-deploy-with-tlse | Error 2026-03-20 16:26:25.842773 | controller | from server (NotFound): pods "watcher-kuttl-api-0" not found' 2026-03-20 16:26:25.842777 | controller | - ' logger.go:42: 16:18:52 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=' 2026-03-20 16:26:25.842783 | controller | - ' logger.go:42: 16:18:53 | watcher-tls-certs-change/1-deploy-with-tlse | running 2026-03-20 16:26:25.842788 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.842792 | controller | - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-20 16:26:25.842796 | controller | cat /etc/pki/tls/certs/public.crt)' 2026-03-20 16:26:25.842800 | controller | - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-20 16:26:25.842804 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-20 16:26:25.842809 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.842813 | controller | - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' 2026-03-20 16:26:25.842817 | controller | - ' exit 1' 2026-03-20 16:26:25.842821 | controller | - ' fi' 2026-03-20 16:26:25.842826 | controller | - ' ' 2026-03-20 16:26:25.842830 | controller | - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-20 16:26:25.842834 | controller | cat /etc/pki/tls/certs/internal.crt)' 2026-03-20 16:26:25.842839 | controller | - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-20 16:26:25.842843 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-20 16:26:25.842848 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.842852 | controller | - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' 2026-03-20 16:26:25.842856 | controller | - ' exit 1' 2026-03-20 16:26:25.842860 | controller | - ' fi' 2026-03-20 16:26:25.842864 | controller | - ' ]' 2026-03-20 16:26:25.842869 | controller | - ' logger.go:42: 16:18:53 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-20 16:26:25.842873 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-20 16:26:25.842877 | controller | - ' logger.go:42: 16:18:53 | watcher-tls-certs-change/1-deploy-with-tlse | Error 2026-03-20 16:26:25.842881 | controller | from server (NotFound): pods "watcher-kuttl-api-0" not found' 2026-03-20 16:26:25.842886 | controller | - ' logger.go:42: 16:18:53 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=' 2026-03-20 16:26:25.842890 | controller | - ' logger.go:42: 16:18:55 | watcher-tls-certs-change/1-deploy-with-tlse | running 2026-03-20 16:26:25.842894 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.842898 | controller | - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-20 16:26:25.842902 | controller | cat /etc/pki/tls/certs/public.crt)' 2026-03-20 16:26:25.842906 | controller | - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-20 16:26:25.842913 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-20 16:26:25.842917 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.842921 | controller | - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' 2026-03-20 16:26:25.842926 | controller | - ' exit 1' 2026-03-20 16:26:25.842932 | controller | - ' fi' 2026-03-20 16:26:25.842937 | controller | - ' ' 2026-03-20 16:26:25.842941 | controller | - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-20 16:26:25.842946 | controller | cat /etc/pki/tls/certs/internal.crt)' 2026-03-20 16:26:25.842950 | controller | - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-20 16:26:25.842954 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-20 16:26:25.842958 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.842963 | controller | - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' 2026-03-20 16:26:25.842967 | controller | - ' exit 1' 2026-03-20 16:26:25.842971 | controller | - ' fi' 2026-03-20 16:26:25.842975 | controller | - ' ]' 2026-03-20 16:26:25.842979 | controller | - ' logger.go:42: 16:18:55 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-20 16:26:25.842992 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-20 16:26:25.842997 | controller | - ' logger.go:42: 16:18:55 | watcher-tls-certs-change/1-deploy-with-tlse | error: 2026-03-20 16:26:25.843001 | controller | unable to upgrade connection: container not found ("watcher-api")' 2026-03-20 16:26:25.843007 | controller | - ' logger.go:42: 16:18:55 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=' 2026-03-20 16:26:25.843012 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | running 2026-03-20 16:26:25.843016 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.843020 | controller | - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-20 16:26:25.843024 | controller | cat /etc/pki/tls/certs/public.crt)' 2026-03-20 16:26:25.843028 | controller | - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-20 16:26:25.843033 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-20 16:26:25.843037 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.843041 | controller | - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' 2026-03-20 16:26:25.843045 | controller | - ' exit 1' 2026-03-20 16:26:25.843049 | controller | - ' fi' 2026-03-20 16:26:25.843054 | controller | - ' ' 2026-03-20 16:26:25.843060 | controller | - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-20 16:26:25.843064 | controller | cat /etc/pki/tls/certs/internal.crt)' 2026-03-20 16:26:25.843068 | controller | - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-20 16:26:25.843072 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-20 16:26:25.843076 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.843081 | controller | - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' 2026-03-20 16:26:25.843085 | controller | - ' exit 1' 2026-03-20 16:26:25.843089 | controller | - ' fi' 2026-03-20 16:26:25.843093 | controller | - ' ]' 2026-03-20 16:26:25.843098 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-20 16:26:25.843102 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-20 16:26:25.843106 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=''-----BEGIN 2026-03-20 16:26:25.843110 | controller | CERTIFICATE-----' 2026-03-20 16:26:25.843115 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY' 2026-03-20 16:26:25.843123 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx' 2026-03-20 16:26:25.843129 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu' 2026-03-20 16:26:25.843133 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH' 2026-03-20 16:26:25.843137 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol' 2026-03-20 16:26:25.843142 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X' 2026-03-20 16:26:25.843146 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI' 2026-03-20 16:26:25.843150 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28' 2026-03-20 16:26:25.843154 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww' 2026-03-20 16:26:25.843158 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/' 2026-03-20 16:26:25.843163 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy' 2026-03-20 16:26:25.843167 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH' 2026-03-20 16:26:25.843171 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp' 2026-03-20 16:26:25.843175 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6' 2026-03-20 16:26:25.843179 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv' 2026-03-20 16:26:25.843191 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl' 2026-03-20 16:26:25.843196 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs' 2026-03-20 16:26:25.843200 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0' 2026-03-20 16:26:25.843204 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9' 2026-03-20 16:26:25.843208 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=' 2026-03-20 16:26:25.843215 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.843219 | controller | CERTIFICATE-----''' 2026-03-20 16:26:25.843223 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 2026-03-20 16:26:25.843227 | controller | --decode' 2026-03-20 16:26:25.843232 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-20 16:26:25.843236 | controller | get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}''' 2026-03-20 16:26:25.843240 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert=''-----BEGIN 2026-03-20 16:26:25.843246 | controller | CERTIFICATE-----' 2026-03-20 16:26:25.843250 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY' 2026-03-20 16:26:25.843254 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx' 2026-03-20 16:26:25.843258 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu' 2026-03-20 16:26:25.843263 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH' 2026-03-20 16:26:25.843267 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol' 2026-03-20 16:26:25.843271 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X' 2026-03-20 16:26:25.843275 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI' 2026-03-20 16:26:25.843280 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28' 2026-03-20 16:26:25.843284 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww' 2026-03-20 16:26:25.843288 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/' 2026-03-20 16:26:25.843292 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy' 2026-03-20 16:26:25.843296 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH' 2026-03-20 16:26:25.843301 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp' 2026-03-20 16:26:25.843305 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6' 2026-03-20 16:26:25.843309 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv' 2026-03-20 16:26:25.843313 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl' 2026-03-20 16:26:25.843317 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs' 2026-03-20 16:26:25.843322 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0' 2026-03-20 16:26:25.843326 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9' 2026-03-20 16:26:25.843338 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=' 2026-03-20 16:26:25.843344 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.843349 | controller | CERTIFICATE-----''' 2026-03-20 16:26:25.843353 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' 2026-03-20 16:26:25.843357 | controller | ''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.843362 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY' 2026-03-20 16:26:25.843369 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx' 2026-03-20 16:26:25.843373 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu' 2026-03-20 16:26:25.843377 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH' 2026-03-20 16:26:25.843382 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol' 2026-03-20 16:26:25.843386 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X' 2026-03-20 16:26:25.843390 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI' 2026-03-20 16:26:25.843394 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28' 2026-03-20 16:26:25.843398 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww' 2026-03-20 16:26:25.843403 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/' 2026-03-20 16:26:25.843407 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy' 2026-03-20 16:26:25.843411 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH' 2026-03-20 16:26:25.843415 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp' 2026-03-20 16:26:25.843419 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6' 2026-03-20 16:26:25.843424 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv' 2026-03-20 16:26:25.843428 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl' 2026-03-20 16:26:25.843432 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs' 2026-03-20 16:26:25.843436 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0' 2026-03-20 16:26:25.843440 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9' 2026-03-20 16:26:25.843445 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=' 2026-03-20 16:26:25.843452 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.843456 | controller | CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.843460 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY' 2026-03-20 16:26:25.843465 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx' 2026-03-20 16:26:25.843469 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu' 2026-03-20 16:26:25.843475 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH' 2026-03-20 16:26:25.843487 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol' 2026-03-20 16:26:25.843491 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X' 2026-03-20 16:26:25.843495 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI' 2026-03-20 16:26:25.843500 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28' 2026-03-20 16:26:25.843504 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww' 2026-03-20 16:26:25.843508 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/' 2026-03-20 16:26:25.843512 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy' 2026-03-20 16:26:25.843516 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH' 2026-03-20 16:26:25.843521 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp' 2026-03-20 16:26:25.843525 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6' 2026-03-20 16:26:25.843529 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv' 2026-03-20 16:26:25.843533 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl' 2026-03-20 16:26:25.843537 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs' 2026-03-20 16:26:25.843542 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0' 2026-03-20 16:26:25.843546 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9' 2026-03-20 16:26:25.843550 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=' 2026-03-20 16:26:25.843556 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.843560 | controller | CERTIFICATE-----'' '']''' 2026-03-20 16:26:25.843565 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-20 16:26:25.843569 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt' 2026-03-20 16:26:25.843573 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert=''-----BEGIN 2026-03-20 16:26:25.843578 | controller | CERTIFICATE-----' 2026-03-20 16:26:25.843582 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw' 2026-03-20 16:26:25.843586 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx' 2026-03-20 16:26:25.843590 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb' 2026-03-20 16:26:25.843596 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2' 2026-03-20 16:26:25.843600 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro' 2026-03-20 16:26:25.843605 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671' 2026-03-20 16:26:25.843609 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B' 2026-03-20 16:26:25.843613 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ' 2026-03-20 16:26:25.843617 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-20 16:26:25.843629 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E' 2026-03-20 16:26:25.843634 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-20 16:26:25.843638 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J' 2026-03-20 16:26:25.843642 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU' 2026-03-20 16:26:25.843647 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq' 2026-03-20 16:26:25.843651 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6' 2026-03-20 16:26:25.843655 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3' 2026-03-20 16:26:25.843659 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk' 2026-03-20 16:26:25.843664 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb' 2026-03-20 16:26:25.843668 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE' 2026-03-20 16:26:25.843672 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==' 2026-03-20 16:26:25.843679 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.843683 | controller | CERTIFICATE-----''' 2026-03-20 16:26:25.843687 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 2026-03-20 16:26:25.843707 | controller | --decode' 2026-03-20 16:26:25.843714 | controller | - ' logger.go:42: 16:18:56 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-20 16:26:25.843718 | controller | get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}''' 2026-03-20 16:26:25.843722 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert=''-----BEGIN 2026-03-20 16:26:25.843726 | controller | CERTIFICATE-----' 2026-03-20 16:26:25.843730 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw' 2026-03-20 16:26:25.843737 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx' 2026-03-20 16:26:25.843741 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb' 2026-03-20 16:26:25.843745 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2' 2026-03-20 16:26:25.843749 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro' 2026-03-20 16:26:25.843753 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671' 2026-03-20 16:26:25.843785 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B' 2026-03-20 16:26:25.843790 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ' 2026-03-20 16:26:25.843794 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-20 16:26:25.843799 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E' 2026-03-20 16:26:25.843803 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-20 16:26:25.843809 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J' 2026-03-20 16:26:25.843814 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU' 2026-03-20 16:26:25.843826 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq' 2026-03-20 16:26:25.843830 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6' 2026-03-20 16:26:25.843835 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3' 2026-03-20 16:26:25.843839 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk' 2026-03-20 16:26:25.843843 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb' 2026-03-20 16:26:25.843847 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE' 2026-03-20 16:26:25.843852 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==' 2026-03-20 16:26:25.843858 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.843862 | controller | CERTIFICATE-----''' 2026-03-20 16:26:25.843866 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' 2026-03-20 16:26:25.843870 | controller | ''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.843875 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw' 2026-03-20 16:26:25.843879 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx' 2026-03-20 16:26:25.843886 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb' 2026-03-20 16:26:25.843890 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2' 2026-03-20 16:26:25.843894 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro' 2026-03-20 16:26:25.843898 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671' 2026-03-20 16:26:25.843902 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B' 2026-03-20 16:26:25.843907 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ' 2026-03-20 16:26:25.843911 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-20 16:26:25.843915 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E' 2026-03-20 16:26:25.843919 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-20 16:26:25.843924 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J' 2026-03-20 16:26:25.843930 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU' 2026-03-20 16:26:25.843934 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq' 2026-03-20 16:26:25.843938 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6' 2026-03-20 16:26:25.843943 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3' 2026-03-20 16:26:25.843947 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk' 2026-03-20 16:26:25.843951 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb' 2026-03-20 16:26:25.843955 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE' 2026-03-20 16:26:25.843965 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==' 2026-03-20 16:26:25.843970 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.843974 | controller | CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.843978 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw' 2026-03-20 16:26:25.843983 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx' 2026-03-20 16:26:25.843987 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb' 2026-03-20 16:26:25.843991 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2' 2026-03-20 16:26:25.843997 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro' 2026-03-20 16:26:25.844002 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671' 2026-03-20 16:26:25.844006 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B' 2026-03-20 16:26:25.844010 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ' 2026-03-20 16:26:25.844014 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-20 16:26:25.844018 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E' 2026-03-20 16:26:25.844023 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-20 16:26:25.844027 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J' 2026-03-20 16:26:25.844031 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU' 2026-03-20 16:26:25.844037 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq' 2026-03-20 16:26:25.844041 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6' 2026-03-20 16:26:25.844046 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3' 2026-03-20 16:26:25.844050 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk' 2026-03-20 16:26:25.844054 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb' 2026-03-20 16:26:25.844058 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE' 2026-03-20 16:26:25.844063 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==' 2026-03-20 16:26:25.844067 | controller | - ' logger.go:42: 16:18:57 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.844071 | controller | CERTIFICATE-----'' '']''' 2026-03-20 16:26:25.844076 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | running 2026-03-20 16:26:25.844080 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.844084 | controller | - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-20 16:26:25.844089 | controller | cat /etc/pki/tls/certs/public.crt)' 2026-03-20 16:26:25.844093 | controller | - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-20 16:26:25.844097 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-20 16:26:25.844101 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.844105 | controller | - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' 2026-03-20 16:26:25.844110 | controller | - ' exit 1' 2026-03-20 16:26:25.844114 | controller | - ' fi' 2026-03-20 16:26:25.844118 | controller | - ' ' 2026-03-20 16:26:25.844122 | controller | - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-20 16:26:25.844126 | controller | cat /etc/pki/tls/certs/internal.crt)' 2026-03-20 16:26:25.844139 | controller | - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-20 16:26:25.844143 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-20 16:26:25.844148 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.844152 | controller | - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' 2026-03-20 16:26:25.844156 | controller | - ' exit 1' 2026-03-20 16:26:25.844160 | controller | - ' fi' 2026-03-20 16:26:25.844165 | controller | - ' ]' 2026-03-20 16:26:25.844169 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-20 16:26:25.844173 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-20 16:26:25.844177 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=''-----BEGIN 2026-03-20 16:26:25.844181 | controller | CERTIFICATE-----' 2026-03-20 16:26:25.844186 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY' 2026-03-20 16:26:25.844190 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx' 2026-03-20 16:26:25.844194 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu' 2026-03-20 16:26:25.844198 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH' 2026-03-20 16:26:25.844203 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol' 2026-03-20 16:26:25.844207 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X' 2026-03-20 16:26:25.844211 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI' 2026-03-20 16:26:25.844215 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28' 2026-03-20 16:26:25.844220 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww' 2026-03-20 16:26:25.844224 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/' 2026-03-20 16:26:25.844228 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy' 2026-03-20 16:26:25.844232 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH' 2026-03-20 16:26:25.844236 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp' 2026-03-20 16:26:25.844240 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6' 2026-03-20 16:26:25.844247 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv' 2026-03-20 16:26:25.844251 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl' 2026-03-20 16:26:25.844255 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs' 2026-03-20 16:26:25.844259 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0' 2026-03-20 16:26:25.844265 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9' 2026-03-20 16:26:25.844270 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=' 2026-03-20 16:26:25.844276 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.844281 | controller | CERTIFICATE-----''' 2026-03-20 16:26:25.844285 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 2026-03-20 16:26:25.844289 | controller | --decode' 2026-03-20 16:26:25.844293 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-20 16:26:25.844298 | controller | get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}''' 2026-03-20 16:26:25.844309 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert=''-----BEGIN 2026-03-20 16:26:25.844313 | controller | CERTIFICATE-----' 2026-03-20 16:26:25.844318 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY' 2026-03-20 16:26:25.844322 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx' 2026-03-20 16:26:25.844326 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu' 2026-03-20 16:26:25.844330 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH' 2026-03-20 16:26:25.844334 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol' 2026-03-20 16:26:25.844338 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X' 2026-03-20 16:26:25.844343 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI' 2026-03-20 16:26:25.844347 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28' 2026-03-20 16:26:25.844351 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww' 2026-03-20 16:26:25.844355 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/' 2026-03-20 16:26:25.844359 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy' 2026-03-20 16:26:25.844363 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH' 2026-03-20 16:26:25.844368 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp' 2026-03-20 16:26:25.844372 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6' 2026-03-20 16:26:25.844376 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv' 2026-03-20 16:26:25.844380 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl' 2026-03-20 16:26:25.844386 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs' 2026-03-20 16:26:25.844391 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0' 2026-03-20 16:26:25.844395 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9' 2026-03-20 16:26:25.844399 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=' 2026-03-20 16:26:25.844405 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.844409 | controller | CERTIFICATE-----''' 2026-03-20 16:26:25.844414 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' 2026-03-20 16:26:25.844418 | controller | ''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.844422 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY' 2026-03-20 16:26:25.844426 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx' 2026-03-20 16:26:25.844430 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu' 2026-03-20 16:26:25.844435 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH' 2026-03-20 16:26:25.844439 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol' 2026-03-20 16:26:25.844450 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X' 2026-03-20 16:26:25.844454 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI' 2026-03-20 16:26:25.844458 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28' 2026-03-20 16:26:25.844462 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww' 2026-03-20 16:26:25.844467 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/' 2026-03-20 16:26:25.844471 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy' 2026-03-20 16:26:25.844475 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH' 2026-03-20 16:26:25.844479 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp' 2026-03-20 16:26:25.844483 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6' 2026-03-20 16:26:25.844488 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv' 2026-03-20 16:26:25.844492 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl' 2026-03-20 16:26:25.844496 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs' 2026-03-20 16:26:25.844502 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0' 2026-03-20 16:26:25.844507 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9' 2026-03-20 16:26:25.844511 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=' 2026-03-20 16:26:25.844538 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.844542 | controller | CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.844547 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY' 2026-03-20 16:26:25.844551 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx' 2026-03-20 16:26:25.844555 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu' 2026-03-20 16:26:25.844559 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH' 2026-03-20 16:26:25.844563 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol' 2026-03-20 16:26:25.844568 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X' 2026-03-20 16:26:25.844572 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI' 2026-03-20 16:26:25.844576 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28' 2026-03-20 16:26:25.844580 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww' 2026-03-20 16:26:25.844584 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/' 2026-03-20 16:26:25.844589 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy' 2026-03-20 16:26:25.844593 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH' 2026-03-20 16:26:25.844603 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp' 2026-03-20 16:26:25.844607 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6' 2026-03-20 16:26:25.844612 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv' 2026-03-20 16:26:25.844616 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl' 2026-03-20 16:26:25.844620 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs' 2026-03-20 16:26:25.844624 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0' 2026-03-20 16:26:25.844628 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9' 2026-03-20 16:26:25.844634 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=' 2026-03-20 16:26:25.844639 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.844643 | controller | CERTIFICATE-----'' '']''' 2026-03-20 16:26:25.844647 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-20 16:26:25.844651 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt' 2026-03-20 16:26:25.844656 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert=''-----BEGIN 2026-03-20 16:26:25.844660 | controller | CERTIFICATE-----' 2026-03-20 16:26:25.844664 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw' 2026-03-20 16:26:25.844668 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx' 2026-03-20 16:26:25.844673 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb' 2026-03-20 16:26:25.844677 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2' 2026-03-20 16:26:25.844681 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro' 2026-03-20 16:26:25.844685 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671' 2026-03-20 16:26:25.844689 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B' 2026-03-20 16:26:25.844708 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ' 2026-03-20 16:26:25.844712 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-20 16:26:25.844716 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E' 2026-03-20 16:26:25.844720 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-20 16:26:25.844725 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J' 2026-03-20 16:26:25.844729 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU' 2026-03-20 16:26:25.844733 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq' 2026-03-20 16:26:25.844737 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6' 2026-03-20 16:26:25.844741 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3' 2026-03-20 16:26:25.844745 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk' 2026-03-20 16:26:25.844755 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb' 2026-03-20 16:26:25.844760 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE' 2026-03-20 16:26:25.844766 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==' 2026-03-20 16:26:25.844772 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.844777 | controller | CERTIFICATE-----''' 2026-03-20 16:26:25.844781 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 2026-03-20 16:26:25.844785 | controller | --decode' 2026-03-20 16:26:25.844789 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-20 16:26:25.844793 | controller | get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}''' 2026-03-20 16:26:25.844798 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert=''-----BEGIN 2026-03-20 16:26:25.844802 | controller | CERTIFICATE-----' 2026-03-20 16:26:25.844806 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw' 2026-03-20 16:26:25.844810 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx' 2026-03-20 16:26:25.844815 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb' 2026-03-20 16:26:25.844819 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2' 2026-03-20 16:26:25.844823 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro' 2026-03-20 16:26:25.844827 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671' 2026-03-20 16:26:25.844831 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B' 2026-03-20 16:26:25.844835 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ' 2026-03-20 16:26:25.844839 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-20 16:26:25.844844 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E' 2026-03-20 16:26:25.844848 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-20 16:26:25.844852 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J' 2026-03-20 16:26:25.844856 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU' 2026-03-20 16:26:25.844860 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq' 2026-03-20 16:26:25.844864 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6' 2026-03-20 16:26:25.844868 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3' 2026-03-20 16:26:25.844873 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk' 2026-03-20 16:26:25.844877 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb' 2026-03-20 16:26:25.844883 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE' 2026-03-20 16:26:25.844887 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==' 2026-03-20 16:26:25.844893 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.844897 | controller | CERTIFICATE-----''' 2026-03-20 16:26:25.844907 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' 2026-03-20 16:26:25.844911 | controller | ''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.844916 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw' 2026-03-20 16:26:25.844920 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx' 2026-03-20 16:26:25.844924 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb' 2026-03-20 16:26:25.844929 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2' 2026-03-20 16:26:25.844933 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro' 2026-03-20 16:26:25.844937 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671' 2026-03-20 16:26:25.844941 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B' 2026-03-20 16:26:25.844945 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ' 2026-03-20 16:26:25.844949 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-20 16:26:25.844954 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E' 2026-03-20 16:26:25.844958 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-20 16:26:25.844962 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J' 2026-03-20 16:26:25.844966 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU' 2026-03-20 16:26:25.844970 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq' 2026-03-20 16:26:25.844974 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6' 2026-03-20 16:26:25.844979 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3' 2026-03-20 16:26:25.844983 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk' 2026-03-20 16:26:25.844987 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb' 2026-03-20 16:26:25.844991 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE' 2026-03-20 16:26:25.844997 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==' 2026-03-20 16:26:25.845004 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.845008 | controller | CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.845012 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw' 2026-03-20 16:26:25.845016 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx' 2026-03-20 16:26:25.845021 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb' 2026-03-20 16:26:25.845025 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2' 2026-03-20 16:26:25.845029 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro' 2026-03-20 16:26:25.845033 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671' 2026-03-20 16:26:25.845043 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B' 2026-03-20 16:26:25.845048 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ' 2026-03-20 16:26:25.845052 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-20 16:26:25.845057 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E' 2026-03-20 16:26:25.845061 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-20 16:26:25.845065 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J' 2026-03-20 16:26:25.845069 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU' 2026-03-20 16:26:25.845073 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq' 2026-03-20 16:26:25.845077 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6' 2026-03-20 16:26:25.845081 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3' 2026-03-20 16:26:25.845086 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk' 2026-03-20 16:26:25.845090 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb' 2026-03-20 16:26:25.845094 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE' 2026-03-20 16:26:25.845098 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==' 2026-03-20 16:26:25.845104 | controller | - ' logger.go:42: 16:18:58 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.845110 | controller | CERTIFICATE-----'' '']''' 2026-03-20 16:26:25.845114 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | running 2026-03-20 16:26:25.845119 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.845123 | controller | - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-20 16:26:25.845127 | controller | cat /etc/pki/tls/certs/public.crt)' 2026-03-20 16:26:25.845131 | controller | - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-20 16:26:25.845135 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-20 16:26:25.845140 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.845144 | controller | - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' 2026-03-20 16:26:25.845148 | controller | - ' exit 1' 2026-03-20 16:26:25.845152 | controller | - ' fi' 2026-03-20 16:26:25.845156 | controller | - ' ' 2026-03-20 16:26:25.845161 | controller | - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-20 16:26:25.845165 | controller | cat /etc/pki/tls/certs/internal.crt)' 2026-03-20 16:26:25.845169 | controller | - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-20 16:26:25.845173 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-20 16:26:25.845177 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.845181 | controller | - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' 2026-03-20 16:26:25.845186 | controller | - ' exit 1' 2026-03-20 16:26:25.845190 | controller | - ' fi' 2026-03-20 16:26:25.845194 | controller | - ' ]' 2026-03-20 16:26:25.845198 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-20 16:26:25.845202 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-20 16:26:25.845207 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=''-----BEGIN 2026-03-20 16:26:25.845211 | controller | CERTIFICATE-----' 2026-03-20 16:26:25.845215 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY' 2026-03-20 16:26:25.845219 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx' 2026-03-20 16:26:25.845223 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu' 2026-03-20 16:26:25.845233 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH' 2026-03-20 16:26:25.845238 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol' 2026-03-20 16:26:25.845244 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X' 2026-03-20 16:26:25.845248 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI' 2026-03-20 16:26:25.845253 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28' 2026-03-20 16:26:25.845257 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww' 2026-03-20 16:26:25.845261 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/' 2026-03-20 16:26:25.845265 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy' 2026-03-20 16:26:25.845272 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH' 2026-03-20 16:26:25.845277 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp' 2026-03-20 16:26:25.845281 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6' 2026-03-20 16:26:25.845285 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv' 2026-03-20 16:26:25.845289 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl' 2026-03-20 16:26:25.845294 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs' 2026-03-20 16:26:25.845298 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0' 2026-03-20 16:26:25.845302 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9' 2026-03-20 16:26:25.845306 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=' 2026-03-20 16:26:25.845312 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.845317 | controller | CERTIFICATE-----''' 2026-03-20 16:26:25.845321 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 2026-03-20 16:26:25.845325 | controller | --decode' 2026-03-20 16:26:25.845329 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-20 16:26:25.845334 | controller | get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}''' 2026-03-20 16:26:25.845338 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert=''-----BEGIN 2026-03-20 16:26:25.845342 | controller | CERTIFICATE-----' 2026-03-20 16:26:25.845346 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY' 2026-03-20 16:26:25.845350 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx' 2026-03-20 16:26:25.845354 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu' 2026-03-20 16:26:25.845359 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH' 2026-03-20 16:26:25.845363 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol' 2026-03-20 16:26:25.845367 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X' 2026-03-20 16:26:25.845371 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI' 2026-03-20 16:26:25.845381 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28' 2026-03-20 16:26:25.845385 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww' 2026-03-20 16:26:25.845390 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/' 2026-03-20 16:26:25.845396 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy' 2026-03-20 16:26:25.845400 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH' 2026-03-20 16:26:25.845404 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp' 2026-03-20 16:26:25.845408 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6' 2026-03-20 16:26:25.845413 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv' 2026-03-20 16:26:25.845417 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl' 2026-03-20 16:26:25.845421 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs' 2026-03-20 16:26:25.845425 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0' 2026-03-20 16:26:25.845429 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9' 2026-03-20 16:26:25.845434 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=' 2026-03-20 16:26:25.845440 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.845444 | controller | CERTIFICATE-----''' 2026-03-20 16:26:25.845448 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' 2026-03-20 16:26:25.845452 | controller | ''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.845457 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY' 2026-03-20 16:26:25.845461 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx' 2026-03-20 16:26:25.845465 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu' 2026-03-20 16:26:25.845469 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH' 2026-03-20 16:26:25.845473 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol' 2026-03-20 16:26:25.845478 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X' 2026-03-20 16:26:25.845482 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI' 2026-03-20 16:26:25.845486 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28' 2026-03-20 16:26:25.845490 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww' 2026-03-20 16:26:25.845494 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/' 2026-03-20 16:26:25.845498 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy' 2026-03-20 16:26:25.845504 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH' 2026-03-20 16:26:25.845509 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp' 2026-03-20 16:26:25.845519 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6' 2026-03-20 16:26:25.845523 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv' 2026-03-20 16:26:25.845527 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl' 2026-03-20 16:26:25.845531 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs' 2026-03-20 16:26:25.845536 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0' 2026-03-20 16:26:25.845540 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9' 2026-03-20 16:26:25.845544 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=' 2026-03-20 16:26:25.845550 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.845554 | controller | CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.845559 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY' 2026-03-20 16:26:25.845563 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx' 2026-03-20 16:26:25.845567 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu' 2026-03-20 16:26:25.845571 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH' 2026-03-20 16:26:25.845575 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol' 2026-03-20 16:26:25.845579 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X' 2026-03-20 16:26:25.845584 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI' 2026-03-20 16:26:25.845588 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28' 2026-03-20 16:26:25.845592 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww' 2026-03-20 16:26:25.845596 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/' 2026-03-20 16:26:25.845601 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy' 2026-03-20 16:26:25.845605 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH' 2026-03-20 16:26:25.845609 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp' 2026-03-20 16:26:25.845615 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6' 2026-03-20 16:26:25.845619 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv' 2026-03-20 16:26:25.845623 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl' 2026-03-20 16:26:25.845627 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs' 2026-03-20 16:26:25.845631 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0' 2026-03-20 16:26:25.845636 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9' 2026-03-20 16:26:25.845640 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=' 2026-03-20 16:26:25.845652 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.845656 | controller | CERTIFICATE-----'' '']''' 2026-03-20 16:26:25.845660 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-20 16:26:25.845665 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt' 2026-03-20 16:26:25.845669 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert=''-----BEGIN 2026-03-20 16:26:25.845673 | controller | CERTIFICATE-----' 2026-03-20 16:26:25.845678 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw' 2026-03-20 16:26:25.845682 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx' 2026-03-20 16:26:25.845686 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb' 2026-03-20 16:26:25.845690 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2' 2026-03-20 16:26:25.845708 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro' 2026-03-20 16:26:25.845712 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671' 2026-03-20 16:26:25.845716 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B' 2026-03-20 16:26:25.845720 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ' 2026-03-20 16:26:25.845724 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-20 16:26:25.845728 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E' 2026-03-20 16:26:25.845733 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-20 16:26:25.845737 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J' 2026-03-20 16:26:25.845741 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU' 2026-03-20 16:26:25.845748 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq' 2026-03-20 16:26:25.845752 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6' 2026-03-20 16:26:25.845756 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3' 2026-03-20 16:26:25.845760 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk' 2026-03-20 16:26:25.845764 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb' 2026-03-20 16:26:25.845768 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE' 2026-03-20 16:26:25.845773 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==' 2026-03-20 16:26:25.845779 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.845784 | controller | CERTIFICATE-----''' 2026-03-20 16:26:25.845788 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 2026-03-20 16:26:25.845792 | controller | --decode' 2026-03-20 16:26:25.845796 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-20 16:26:25.845801 | controller | get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}''' 2026-03-20 16:26:25.845805 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert=''-----BEGIN 2026-03-20 16:26:25.845809 | controller | CERTIFICATE-----' 2026-03-20 16:26:25.845813 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw' 2026-03-20 16:26:25.845824 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx' 2026-03-20 16:26:25.845828 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb' 2026-03-20 16:26:25.845833 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2' 2026-03-20 16:26:25.845837 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro' 2026-03-20 16:26:25.845841 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671' 2026-03-20 16:26:25.845845 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B' 2026-03-20 16:26:25.845849 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ' 2026-03-20 16:26:25.845854 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-20 16:26:25.845858 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E' 2026-03-20 16:26:25.845862 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-20 16:26:25.845868 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J' 2026-03-20 16:26:25.845872 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU' 2026-03-20 16:26:25.845876 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq' 2026-03-20 16:26:25.845881 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6' 2026-03-20 16:26:25.845885 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3' 2026-03-20 16:26:25.845889 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk' 2026-03-20 16:26:25.845893 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb' 2026-03-20 16:26:25.845897 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE' 2026-03-20 16:26:25.845902 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==' 2026-03-20 16:26:25.845908 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.845912 | controller | CERTIFICATE-----''' 2026-03-20 16:26:25.845916 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' 2026-03-20 16:26:25.845921 | controller | ''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.845925 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw' 2026-03-20 16:26:25.845929 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx' 2026-03-20 16:26:25.845933 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb' 2026-03-20 16:26:25.845937 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2' 2026-03-20 16:26:25.845942 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro' 2026-03-20 16:26:25.845946 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671' 2026-03-20 16:26:25.845950 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B' 2026-03-20 16:26:25.845960 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ' 2026-03-20 16:26:25.845965 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-20 16:26:25.845969 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E' 2026-03-20 16:26:25.845973 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-20 16:26:25.845978 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J' 2026-03-20 16:26:25.845983 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU' 2026-03-20 16:26:25.845988 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq' 2026-03-20 16:26:25.845992 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6' 2026-03-20 16:26:25.845996 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3' 2026-03-20 16:26:25.846000 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk' 2026-03-20 16:26:25.846004 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb' 2026-03-20 16:26:25.846009 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE' 2026-03-20 16:26:25.846013 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==' 2026-03-20 16:26:25.846019 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.846023 | controller | CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.846027 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw' 2026-03-20 16:26:25.846031 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx' 2026-03-20 16:26:25.846036 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb' 2026-03-20 16:26:25.846040 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2' 2026-03-20 16:26:25.846044 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro' 2026-03-20 16:26:25.846048 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671' 2026-03-20 16:26:25.846052 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B' 2026-03-20 16:26:25.846056 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ' 2026-03-20 16:26:25.846061 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-20 16:26:25.846065 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E' 2026-03-20 16:26:25.846069 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-20 16:26:25.846073 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J' 2026-03-20 16:26:25.846077 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU' 2026-03-20 16:26:25.846088 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq' 2026-03-20 16:26:25.846094 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6' 2026-03-20 16:26:25.846098 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3' 2026-03-20 16:26:25.846102 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk' 2026-03-20 16:26:25.846107 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb' 2026-03-20 16:26:25.846111 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE' 2026-03-20 16:26:25.846115 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==' 2026-03-20 16:26:25.846122 | controller | - ' logger.go:42: 16:19:00 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.846126 | controller | CERTIFICATE-----'' '']''' 2026-03-20 16:26:25.846131 | controller | - ' logger.go:42: 16:19:01 | watcher-tls-certs-change/1-deploy-with-tlse | running 2026-03-20 16:26:25.846135 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.846139 | controller | - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-20 16:26:25.846144 | controller | cat /etc/pki/tls/certs/public.crt)' 2026-03-20 16:26:25.846148 | controller | - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-20 16:26:25.846152 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-20 16:26:25.846156 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.846160 | controller | - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' 2026-03-20 16:26:25.846165 | controller | - ' exit 1' 2026-03-20 16:26:25.846169 | controller | - ' fi' 2026-03-20 16:26:25.846173 | controller | - ' ' 2026-03-20 16:26:25.846177 | controller | - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-20 16:26:25.846183 | controller | cat /etc/pki/tls/certs/internal.crt)' 2026-03-20 16:26:25.846188 | controller | - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-20 16:26:25.846192 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-20 16:26:25.846196 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.846200 | controller | - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' 2026-03-20 16:26:25.846204 | controller | - ' exit 1' 2026-03-20 16:26:25.846209 | controller | - ' fi' 2026-03-20 16:26:25.846213 | controller | - ' ]' 2026-03-20 16:26:25.846217 | controller | - ' logger.go:42: 16:19:01 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-20 16:26:25.846221 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-20 16:26:25.846225 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=''-----BEGIN 2026-03-20 16:26:25.846230 | controller | CERTIFICATE-----' 2026-03-20 16:26:25.846234 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY' 2026-03-20 16:26:25.846238 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx' 2026-03-20 16:26:25.846242 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu' 2026-03-20 16:26:25.846246 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH' 2026-03-20 16:26:25.846253 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol' 2026-03-20 16:26:25.846257 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X' 2026-03-20 16:26:25.846261 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI' 2026-03-20 16:26:25.846265 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28' 2026-03-20 16:26:25.846270 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww' 2026-03-20 16:26:25.846274 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/' 2026-03-20 16:26:25.846285 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy' 2026-03-20 16:26:25.846289 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH' 2026-03-20 16:26:25.846294 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp' 2026-03-20 16:26:25.846298 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6' 2026-03-20 16:26:25.846302 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv' 2026-03-20 16:26:25.846306 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl' 2026-03-20 16:26:25.846311 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs' 2026-03-20 16:26:25.846317 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0' 2026-03-20 16:26:25.846321 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9' 2026-03-20 16:26:25.846325 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=' 2026-03-20 16:26:25.846339 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.846344 | controller | CERTIFICATE-----''' 2026-03-20 16:26:25.846348 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 2026-03-20 16:26:25.846352 | controller | --decode' 2026-03-20 16:26:25.846357 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-20 16:26:25.846361 | controller | get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}''' 2026-03-20 16:26:25.846365 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert=''-----BEGIN 2026-03-20 16:26:25.846369 | controller | CERTIFICATE-----' 2026-03-20 16:26:25.846373 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY' 2026-03-20 16:26:25.846378 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx' 2026-03-20 16:26:25.846382 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu' 2026-03-20 16:26:25.846388 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH' 2026-03-20 16:26:25.846392 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol' 2026-03-20 16:26:25.846397 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X' 2026-03-20 16:26:25.846401 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI' 2026-03-20 16:26:25.846405 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28' 2026-03-20 16:26:25.846409 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww' 2026-03-20 16:26:25.846413 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/' 2026-03-20 16:26:25.846418 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy' 2026-03-20 16:26:25.846422 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH' 2026-03-20 16:26:25.846426 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp' 2026-03-20 16:26:25.846430 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6' 2026-03-20 16:26:25.846440 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv' 2026-03-20 16:26:25.846444 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl' 2026-03-20 16:26:25.846448 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs' 2026-03-20 16:26:25.846453 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0' 2026-03-20 16:26:25.846457 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9' 2026-03-20 16:26:25.846461 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=' 2026-03-20 16:26:25.846466 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.846470 | controller | CERTIFICATE-----''' 2026-03-20 16:26:25.846474 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' 2026-03-20 16:26:25.846480 | controller | ''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.846485 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY' 2026-03-20 16:26:25.846489 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx' 2026-03-20 16:26:25.846493 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu' 2026-03-20 16:26:25.846497 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH' 2026-03-20 16:26:25.846503 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol' 2026-03-20 16:26:25.846507 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X' 2026-03-20 16:26:25.846511 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI' 2026-03-20 16:26:25.846516 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28' 2026-03-20 16:26:25.846520 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww' 2026-03-20 16:26:25.846524 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/' 2026-03-20 16:26:25.846528 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy' 2026-03-20 16:26:25.846532 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH' 2026-03-20 16:26:25.846536 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp' 2026-03-20 16:26:25.846541 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6' 2026-03-20 16:26:25.846545 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv' 2026-03-20 16:26:25.846549 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl' 2026-03-20 16:26:25.846553 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs' 2026-03-20 16:26:25.846557 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0' 2026-03-20 16:26:25.846561 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9' 2026-03-20 16:26:25.846566 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=' 2026-03-20 16:26:25.846572 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.846576 | controller | CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.846586 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY' 2026-03-20 16:26:25.846590 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx' 2026-03-20 16:26:25.846595 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu' 2026-03-20 16:26:25.846599 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH' 2026-03-20 16:26:25.846603 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol' 2026-03-20 16:26:25.846607 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X' 2026-03-20 16:26:25.846613 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI' 2026-03-20 16:26:25.846617 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28' 2026-03-20 16:26:25.846622 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww' 2026-03-20 16:26:25.846626 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/' 2026-03-20 16:26:25.846630 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy' 2026-03-20 16:26:25.846634 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH' 2026-03-20 16:26:25.846638 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp' 2026-03-20 16:26:25.846643 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6' 2026-03-20 16:26:25.846647 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv' 2026-03-20 16:26:25.846651 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl' 2026-03-20 16:26:25.846655 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs' 2026-03-20 16:26:25.846660 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0' 2026-03-20 16:26:25.846664 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9' 2026-03-20 16:26:25.846670 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=' 2026-03-20 16:26:25.846675 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.846679 | controller | CERTIFICATE-----'' '']''' 2026-03-20 16:26:25.846684 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-20 16:26:25.846688 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt' 2026-03-20 16:26:25.846703 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert=''-----BEGIN 2026-03-20 16:26:25.846709 | controller | CERTIFICATE-----' 2026-03-20 16:26:25.846713 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw' 2026-03-20 16:26:25.846718 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx' 2026-03-20 16:26:25.846722 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb' 2026-03-20 16:26:25.846726 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2' 2026-03-20 16:26:25.846736 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro' 2026-03-20 16:26:25.846741 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671' 2026-03-20 16:26:25.846747 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B' 2026-03-20 16:26:25.846752 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ' 2026-03-20 16:26:25.846756 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-20 16:26:25.846760 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E' 2026-03-20 16:26:25.846764 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-20 16:26:25.846768 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J' 2026-03-20 16:26:25.846772 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU' 2026-03-20 16:26:25.846777 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq' 2026-03-20 16:26:25.846781 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6' 2026-03-20 16:26:25.846785 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3' 2026-03-20 16:26:25.846789 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk' 2026-03-20 16:26:25.846794 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb' 2026-03-20 16:26:25.846798 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE' 2026-03-20 16:26:25.846802 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==' 2026-03-20 16:26:25.846808 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.846812 | controller | CERTIFICATE-----''' 2026-03-20 16:26:25.846817 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-20 16:26:25.846821 | controller | get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}''' 2026-03-20 16:26:25.846825 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 2026-03-20 16:26:25.846829 | controller | --decode' 2026-03-20 16:26:25.846834 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert=''-----BEGIN 2026-03-20 16:26:25.846838 | controller | CERTIFICATE-----' 2026-03-20 16:26:25.846842 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw' 2026-03-20 16:26:25.846846 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx' 2026-03-20 16:26:25.846850 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb' 2026-03-20 16:26:25.846855 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2' 2026-03-20 16:26:25.846859 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro' 2026-03-20 16:26:25.846865 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671' 2026-03-20 16:26:25.846869 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B' 2026-03-20 16:26:25.846873 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ' 2026-03-20 16:26:25.846884 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-20 16:26:25.846888 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E' 2026-03-20 16:26:25.846892 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-20 16:26:25.846897 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J' 2026-03-20 16:26:25.846901 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU' 2026-03-20 16:26:25.846905 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq' 2026-03-20 16:26:25.846909 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6' 2026-03-20 16:26:25.846914 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3' 2026-03-20 16:26:25.846918 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk' 2026-03-20 16:26:25.846922 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb' 2026-03-20 16:26:25.846947 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE' 2026-03-20 16:26:25.846952 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==' 2026-03-20 16:26:25.846960 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.846964 | controller | CERTIFICATE-----''' 2026-03-20 16:26:25.846968 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' 2026-03-20 16:26:25.846973 | controller | ''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.846977 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw' 2026-03-20 16:26:25.846981 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx' 2026-03-20 16:26:25.846985 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb' 2026-03-20 16:26:25.846989 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2' 2026-03-20 16:26:25.846994 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro' 2026-03-20 16:26:25.846998 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671' 2026-03-20 16:26:25.847004 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B' 2026-03-20 16:26:25.847008 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ' 2026-03-20 16:26:25.847012 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-20 16:26:25.847017 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E' 2026-03-20 16:26:25.847021 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-20 16:26:25.847025 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J' 2026-03-20 16:26:25.847029 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU' 2026-03-20 16:26:25.847033 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq' 2026-03-20 16:26:25.847044 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6' 2026-03-20 16:26:25.847048 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3' 2026-03-20 16:26:25.847053 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk' 2026-03-20 16:26:25.847057 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb' 2026-03-20 16:26:25.847061 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE' 2026-03-20 16:26:25.847065 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==' 2026-03-20 16:26:25.847072 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.847076 | controller | CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.847081 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw' 2026-03-20 16:26:25.847085 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx' 2026-03-20 16:26:25.847089 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb' 2026-03-20 16:26:25.847093 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2' 2026-03-20 16:26:25.847097 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro' 2026-03-20 16:26:25.847101 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671' 2026-03-20 16:26:25.847106 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B' 2026-03-20 16:26:25.847110 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ' 2026-03-20 16:26:25.847116 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-20 16:26:25.847120 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E' 2026-03-20 16:26:25.847124 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-20 16:26:25.847128 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J' 2026-03-20 16:26:25.847133 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU' 2026-03-20 16:26:25.847137 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq' 2026-03-20 16:26:25.847141 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6' 2026-03-20 16:26:25.847145 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3' 2026-03-20 16:26:25.847149 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk' 2026-03-20 16:26:25.847154 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb' 2026-03-20 16:26:25.847158 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE' 2026-03-20 16:26:25.847162 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==' 2026-03-20 16:26:25.847168 | controller | - ' logger.go:42: 16:19:02 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.847173 | controller | CERTIFICATE-----'' '']''' 2026-03-20 16:26:25.847182 | controller | - ' logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | running 2026-03-20 16:26:25.847187 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.847191 | controller | - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-20 16:26:25.847196 | controller | cat /etc/pki/tls/certs/public.crt)' 2026-03-20 16:26:25.847200 | controller | - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-20 16:26:25.847204 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-20 16:26:25.847208 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.847213 | controller | - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' 2026-03-20 16:26:25.847217 | controller | - ' exit 1' 2026-03-20 16:26:25.847221 | controller | - ' fi' 2026-03-20 16:26:25.847225 | controller | - ' ' 2026-03-20 16:26:25.847229 | controller | - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-20 16:26:25.847234 | controller | cat /etc/pki/tls/certs/internal.crt)' 2026-03-20 16:26:25.847238 | controller | - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-20 16:26:25.847242 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-20 16:26:25.847246 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.847250 | controller | - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' 2026-03-20 16:26:25.847255 | controller | - ' exit 1' 2026-03-20 16:26:25.847259 | controller | - ' fi' 2026-03-20 16:26:25.847263 | controller | - ' ]' 2026-03-20 16:26:25.847269 | controller | - ' logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-20 16:26:25.847274 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-20 16:26:25.847278 | controller | - ' logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=''-----BEGIN 2026-03-20 16:26:25.847282 | controller | CERTIFICATE-----' 2026-03-20 16:26:25.847286 | controller | - ' logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY' 2026-03-20 16:26:25.847291 | controller | - ' logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx' 2026-03-20 16:26:25.847295 | controller | - ' logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu' 2026-03-20 16:26:25.847299 | controller | - ' logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH' 2026-03-20 16:26:25.847303 | controller | - ' logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol' 2026-03-20 16:26:25.847308 | controller | - ' logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X' 2026-03-20 16:26:25.847312 | controller | - ' logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI' 2026-03-20 16:26:25.847316 | controller | - ' logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28' 2026-03-20 16:26:25.847322 | controller | - ' logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww' 2026-03-20 16:26:25.847327 | controller | - ' logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/' 2026-03-20 16:26:25.847331 | controller | - ' logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy' 2026-03-20 16:26:25.847335 | controller | - ' logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH' 2026-03-20 16:26:25.847339 | controller | - ' logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp' 2026-03-20 16:26:25.847535 | controller | - ' logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6' 2026-03-20 16:26:25.847543 | controller | - ' logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv' 2026-03-20 16:26:25.847547 | controller | - ' logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl' 2026-03-20 16:26:25.847551 | controller | - ' logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs' 2026-03-20 16:26:25.847556 | controller | - ' logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0' 2026-03-20 16:26:25.847569 | controller | - ' logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9' 2026-03-20 16:26:25.847573 | controller | - ' logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=' 2026-03-20 16:26:25.847579 | controller | - ' logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.847586 | controller | CERTIFICATE-----''' 2026-03-20 16:26:25.847590 | controller | - ' logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 2026-03-20 16:26:25.847595 | controller | --decode' 2026-03-20 16:26:25.847599 | controller | - ' logger.go:42: 16:19:03 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-20 16:26:25.847603 | controller | get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}''' 2026-03-20 16:26:25.847608 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert=''-----BEGIN 2026-03-20 16:26:25.847612 | controller | CERTIFICATE-----' 2026-03-20 16:26:25.847616 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY' 2026-03-20 16:26:25.847620 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx' 2026-03-20 16:26:25.847625 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu' 2026-03-20 16:26:25.847629 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH' 2026-03-20 16:26:25.847633 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol' 2026-03-20 16:26:25.847638 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X' 2026-03-20 16:26:25.847642 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI' 2026-03-20 16:26:25.847646 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28' 2026-03-20 16:26:25.847650 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww' 2026-03-20 16:26:25.847655 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/' 2026-03-20 16:26:25.847659 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy' 2026-03-20 16:26:25.847663 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH' 2026-03-20 16:26:25.847667 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp' 2026-03-20 16:26:25.847671 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6' 2026-03-20 16:26:25.847676 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv' 2026-03-20 16:26:25.847680 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl' 2026-03-20 16:26:25.847684 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs' 2026-03-20 16:26:25.847689 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0' 2026-03-20 16:26:25.847704 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9' 2026-03-20 16:26:25.847712 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=' 2026-03-20 16:26:25.847720 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.847725 | controller | CERTIFICATE-----''' 2026-03-20 16:26:25.847729 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' 2026-03-20 16:26:25.847733 | controller | ''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.847744 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY' 2026-03-20 16:26:25.847749 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx' 2026-03-20 16:26:25.847753 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu' 2026-03-20 16:26:25.847758 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH' 2026-03-20 16:26:25.847792 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol' 2026-03-20 16:26:25.847798 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X' 2026-03-20 16:26:25.847803 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI' 2026-03-20 16:26:25.847808 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28' 2026-03-20 16:26:25.847813 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww' 2026-03-20 16:26:25.847818 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/' 2026-03-20 16:26:25.847822 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy' 2026-03-20 16:26:25.847827 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH' 2026-03-20 16:26:25.847833 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp' 2026-03-20 16:26:25.847838 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6' 2026-03-20 16:26:25.847842 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv' 2026-03-20 16:26:25.847847 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl' 2026-03-20 16:26:25.847852 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs' 2026-03-20 16:26:25.847857 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0' 2026-03-20 16:26:25.847862 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9' 2026-03-20 16:26:25.847867 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=' 2026-03-20 16:26:25.847875 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.847883 | controller | CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.847888 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY' 2026-03-20 16:26:25.847893 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx' 2026-03-20 16:26:25.847898 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu' 2026-03-20 16:26:25.847903 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH' 2026-03-20 16:26:25.847908 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol' 2026-03-20 16:26:25.847913 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X' 2026-03-20 16:26:25.847918 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI' 2026-03-20 16:26:25.847931 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28' 2026-03-20 16:26:25.847936 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww' 2026-03-20 16:26:25.847941 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/' 2026-03-20 16:26:25.847946 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy' 2026-03-20 16:26:25.847951 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH' 2026-03-20 16:26:25.847956 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp' 2026-03-20 16:26:25.847961 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6' 2026-03-20 16:26:25.847966 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv' 2026-03-20 16:26:25.847971 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl' 2026-03-20 16:26:25.847976 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs' 2026-03-20 16:26:25.847981 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0' 2026-03-20 16:26:25.847986 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9' 2026-03-20 16:26:25.847990 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=' 2026-03-20 16:26:25.847998 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.848003 | controller | CERTIFICATE-----'' '']''' 2026-03-20 16:26:25.848008 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-20 16:26:25.848013 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt' 2026-03-20 16:26:25.848018 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert=''-----BEGIN 2026-03-20 16:26:25.848025 | controller | CERTIFICATE-----' 2026-03-20 16:26:25.848029 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw' 2026-03-20 16:26:25.848034 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx' 2026-03-20 16:26:25.848038 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb' 2026-03-20 16:26:25.848042 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2' 2026-03-20 16:26:25.848046 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro' 2026-03-20 16:26:25.848051 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671' 2026-03-20 16:26:25.848055 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B' 2026-03-20 16:26:25.848059 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ' 2026-03-20 16:26:25.848063 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-20 16:26:25.848068 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E' 2026-03-20 16:26:25.848072 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-20 16:26:25.848076 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J' 2026-03-20 16:26:25.848087 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU' 2026-03-20 16:26:25.848092 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq' 2026-03-20 16:26:25.848096 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6' 2026-03-20 16:26:25.848100 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3' 2026-03-20 16:26:25.848104 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk' 2026-03-20 16:26:25.848109 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb' 2026-03-20 16:26:25.848113 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE' 2026-03-20 16:26:25.848117 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==' 2026-03-20 16:26:25.848124 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.848128 | controller | CERTIFICATE-----''' 2026-03-20 16:26:25.848132 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 2026-03-20 16:26:25.848136 | controller | --decode' 2026-03-20 16:26:25.848141 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-20 16:26:25.848155 | controller | get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}''' 2026-03-20 16:26:25.848160 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert=''-----BEGIN 2026-03-20 16:26:25.848164 | controller | CERTIFICATE-----' 2026-03-20 16:26:25.848168 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw' 2026-03-20 16:26:25.848172 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx' 2026-03-20 16:26:25.848177 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb' 2026-03-20 16:26:25.848181 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2' 2026-03-20 16:26:25.848185 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro' 2026-03-20 16:26:25.848189 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671' 2026-03-20 16:26:25.848194 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B' 2026-03-20 16:26:25.848198 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ' 2026-03-20 16:26:25.848202 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-20 16:26:25.848206 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E' 2026-03-20 16:26:25.848210 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-20 16:26:25.848214 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J' 2026-03-20 16:26:25.848219 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU' 2026-03-20 16:26:25.848223 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq' 2026-03-20 16:26:25.848227 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6' 2026-03-20 16:26:25.848231 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3' 2026-03-20 16:26:25.848243 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk' 2026-03-20 16:26:25.848247 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb' 2026-03-20 16:26:25.848251 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE' 2026-03-20 16:26:25.848256 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==' 2026-03-20 16:26:25.848262 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.848266 | controller | CERTIFICATE-----''' 2026-03-20 16:26:25.848272 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' 2026-03-20 16:26:25.848276 | controller | ''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.848281 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw' 2026-03-20 16:26:25.848285 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx' 2026-03-20 16:26:25.848289 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb' 2026-03-20 16:26:25.848294 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2' 2026-03-20 16:26:25.848298 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro' 2026-03-20 16:26:25.848302 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671' 2026-03-20 16:26:25.848306 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B' 2026-03-20 16:26:25.848310 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ' 2026-03-20 16:26:25.848315 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-20 16:26:25.848319 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E' 2026-03-20 16:26:25.848324 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-20 16:26:25.848329 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J' 2026-03-20 16:26:25.848334 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU' 2026-03-20 16:26:25.848339 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq' 2026-03-20 16:26:25.848344 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6' 2026-03-20 16:26:25.848349 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3' 2026-03-20 16:26:25.848353 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk' 2026-03-20 16:26:25.848358 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb' 2026-03-20 16:26:25.848362 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE' 2026-03-20 16:26:25.848366 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==' 2026-03-20 16:26:25.848373 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.848377 | controller | CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.848381 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw' 2026-03-20 16:26:25.848394 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx' 2026-03-20 16:26:25.848399 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb' 2026-03-20 16:26:25.848403 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2' 2026-03-20 16:26:25.848407 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro' 2026-03-20 16:26:25.848411 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671' 2026-03-20 16:26:25.848416 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B' 2026-03-20 16:26:25.848420 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ' 2026-03-20 16:26:25.848424 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-20 16:26:25.848428 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E' 2026-03-20 16:26:25.848433 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-20 16:26:25.848437 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J' 2026-03-20 16:26:25.848441 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU' 2026-03-20 16:26:25.848445 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq' 2026-03-20 16:26:25.848449 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6' 2026-03-20 16:26:25.848453 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3' 2026-03-20 16:26:25.848458 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk' 2026-03-20 16:26:25.848462 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb' 2026-03-20 16:26:25.848466 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE' 2026-03-20 16:26:25.848472 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==' 2026-03-20 16:26:25.848478 | controller | - ' logger.go:42: 16:19:04 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.848483 | controller | CERTIFICATE-----'' '']''' 2026-03-20 16:26:25.848487 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | running 2026-03-20 16:26:25.848491 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.848495 | controller | - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-20 16:26:25.848500 | controller | cat /etc/pki/tls/certs/public.crt)' 2026-03-20 16:26:25.848504 | controller | - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-20 16:26:25.848508 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-20 16:26:25.848514 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.848519 | controller | - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' 2026-03-20 16:26:25.848523 | controller | - ' exit 1' 2026-03-20 16:26:25.848527 | controller | - ' fi' 2026-03-20 16:26:25.848531 | controller | - ' ' 2026-03-20 16:26:25.848536 | controller | - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-20 16:26:25.848540 | controller | cat /etc/pki/tls/certs/internal.crt)' 2026-03-20 16:26:25.848544 | controller | - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-20 16:26:25.848548 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-20 16:26:25.848552 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.848557 | controller | - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' 2026-03-20 16:26:25.848561 | controller | - ' exit 1' 2026-03-20 16:26:25.848565 | controller | - ' fi' 2026-03-20 16:26:25.848569 | controller | - ' ]' 2026-03-20 16:26:25.848573 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-20 16:26:25.848584 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-20 16:26:25.848588 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=''-----BEGIN 2026-03-20 16:26:25.848592 | controller | CERTIFICATE-----' 2026-03-20 16:26:25.848597 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY' 2026-03-20 16:26:25.848601 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx' 2026-03-20 16:26:25.848605 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu' 2026-03-20 16:26:25.848610 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH' 2026-03-20 16:26:25.848614 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol' 2026-03-20 16:26:25.848618 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X' 2026-03-20 16:26:25.848622 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI' 2026-03-20 16:26:25.848626 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28' 2026-03-20 16:26:25.848631 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww' 2026-03-20 16:26:25.848635 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/' 2026-03-20 16:26:25.848639 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy' 2026-03-20 16:26:25.848643 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH' 2026-03-20 16:26:25.848647 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp' 2026-03-20 16:26:25.848652 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6' 2026-03-20 16:26:25.848656 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv' 2026-03-20 16:26:25.848662 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl' 2026-03-20 16:26:25.848667 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs' 2026-03-20 16:26:25.848671 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0' 2026-03-20 16:26:25.848675 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9' 2026-03-20 16:26:25.848679 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=' 2026-03-20 16:26:25.848683 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.848688 | controller | CERTIFICATE-----''' 2026-03-20 16:26:25.848703 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-20 16:26:25.848710 | controller | get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}''' 2026-03-20 16:26:25.848714 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 2026-03-20 16:26:25.848719 | controller | --decode' 2026-03-20 16:26:25.848723 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert=''-----BEGIN 2026-03-20 16:26:25.848727 | controller | CERTIFICATE-----' 2026-03-20 16:26:25.848731 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY' 2026-03-20 16:26:25.848736 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx' 2026-03-20 16:26:25.848746 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu' 2026-03-20 16:26:25.848751 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH' 2026-03-20 16:26:25.848755 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol' 2026-03-20 16:26:25.848760 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X' 2026-03-20 16:26:25.848764 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI' 2026-03-20 16:26:25.848768 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28' 2026-03-20 16:26:25.848772 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww' 2026-03-20 16:26:25.848776 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/' 2026-03-20 16:26:25.848780 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy' 2026-03-20 16:26:25.848785 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH' 2026-03-20 16:26:25.848789 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp' 2026-03-20 16:26:25.848795 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6' 2026-03-20 16:26:25.848800 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv' 2026-03-20 16:26:25.848804 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl' 2026-03-20 16:26:25.848808 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs' 2026-03-20 16:26:25.848812 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0' 2026-03-20 16:26:25.848816 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9' 2026-03-20 16:26:25.848820 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=' 2026-03-20 16:26:25.848827 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.848831 | controller | CERTIFICATE-----''' 2026-03-20 16:26:25.848835 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' 2026-03-20 16:26:25.848840 | controller | ''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.848844 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY' 2026-03-20 16:26:25.848848 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx' 2026-03-20 16:26:25.848852 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu' 2026-03-20 16:26:25.848857 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH' 2026-03-20 16:26:25.848861 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol' 2026-03-20 16:26:25.848865 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X' 2026-03-20 16:26:25.848869 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI' 2026-03-20 16:26:25.848873 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28' 2026-03-20 16:26:25.848883 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww' 2026-03-20 16:26:25.848888 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/' 2026-03-20 16:26:25.848892 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy' 2026-03-20 16:26:25.848896 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH' 2026-03-20 16:26:25.848900 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp' 2026-03-20 16:26:25.848905 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6' 2026-03-20 16:26:25.848909 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv' 2026-03-20 16:26:25.848916 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl' 2026-03-20 16:26:25.848921 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs' 2026-03-20 16:26:25.848925 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0' 2026-03-20 16:26:25.848929 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9' 2026-03-20 16:26:25.848933 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=' 2026-03-20 16:26:25.848943 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.848947 | controller | CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.848952 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY' 2026-03-20 16:26:25.848956 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx' 2026-03-20 16:26:25.848960 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu' 2026-03-20 16:26:25.848964 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH' 2026-03-20 16:26:25.848968 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol' 2026-03-20 16:26:25.848973 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X' 2026-03-20 16:26:25.848977 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI' 2026-03-20 16:26:25.848981 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28' 2026-03-20 16:26:25.848985 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww' 2026-03-20 16:26:25.848990 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/' 2026-03-20 16:26:25.848994 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy' 2026-03-20 16:26:25.848998 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH' 2026-03-20 16:26:25.849002 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp' 2026-03-20 16:26:25.849006 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6' 2026-03-20 16:26:25.849011 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv' 2026-03-20 16:26:25.849021 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl' 2026-03-20 16:26:25.849026 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs' 2026-03-20 16:26:25.849032 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0' 2026-03-20 16:26:25.849036 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9' 2026-03-20 16:26:25.849041 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=' 2026-03-20 16:26:25.849047 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.849051 | controller | CERTIFICATE-----'' '']''' 2026-03-20 16:26:25.849055 | controller | - ' logger.go:42: 16:19:05 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-20 16:26:25.849059 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt' 2026-03-20 16:26:25.849064 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert=''-----BEGIN 2026-03-20 16:26:25.849068 | controller | CERTIFICATE-----' 2026-03-20 16:26:25.849072 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw' 2026-03-20 16:26:25.849076 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx' 2026-03-20 16:26:25.849080 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb' 2026-03-20 16:26:25.849085 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2' 2026-03-20 16:26:25.849089 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro' 2026-03-20 16:26:25.849093 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671' 2026-03-20 16:26:25.849097 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B' 2026-03-20 16:26:25.849101 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ' 2026-03-20 16:26:25.849106 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-20 16:26:25.849110 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E' 2026-03-20 16:26:25.849114 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-20 16:26:25.849118 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J' 2026-03-20 16:26:25.849122 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU' 2026-03-20 16:26:25.849127 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq' 2026-03-20 16:26:25.849131 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6' 2026-03-20 16:26:25.849135 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3' 2026-03-20 16:26:25.849141 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk' 2026-03-20 16:26:25.849145 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb' 2026-03-20 16:26:25.849150 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE' 2026-03-20 16:26:25.849154 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==' 2026-03-20 16:26:25.849173 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.849177 | controller | CERTIFICATE-----''' 2026-03-20 16:26:25.849182 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-20 16:26:25.849186 | controller | get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}''' 2026-03-20 16:26:25.849190 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 2026-03-20 16:26:25.849195 | controller | --decode' 2026-03-20 16:26:25.849199 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert=''-----BEGIN 2026-03-20 16:26:25.849203 | controller | CERTIFICATE-----' 2026-03-20 16:26:25.849207 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw' 2026-03-20 16:26:25.849211 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx' 2026-03-20 16:26:25.849215 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb' 2026-03-20 16:26:25.849220 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2' 2026-03-20 16:26:25.849224 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro' 2026-03-20 16:26:25.849228 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671' 2026-03-20 16:26:25.849232 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B' 2026-03-20 16:26:25.849236 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ' 2026-03-20 16:26:25.849241 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-20 16:26:25.849245 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E' 2026-03-20 16:26:25.849249 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-20 16:26:25.849253 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J' 2026-03-20 16:26:25.849257 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU' 2026-03-20 16:26:25.849262 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq' 2026-03-20 16:26:25.849266 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6' 2026-03-20 16:26:25.849272 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3' 2026-03-20 16:26:25.849276 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk' 2026-03-20 16:26:25.849280 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb' 2026-03-20 16:26:25.849284 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE' 2026-03-20 16:26:25.849288 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==' 2026-03-20 16:26:25.849301 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.849306 | controller | CERTIFICATE-----''' 2026-03-20 16:26:25.849310 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' 2026-03-20 16:26:25.849314 | controller | ''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.849318 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw' 2026-03-20 16:26:25.849322 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx' 2026-03-20 16:26:25.849333 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb' 2026-03-20 16:26:25.849337 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2' 2026-03-20 16:26:25.849341 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro' 2026-03-20 16:26:25.849345 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671' 2026-03-20 16:26:25.849350 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B' 2026-03-20 16:26:25.849354 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ' 2026-03-20 16:26:25.849358 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-20 16:26:25.849362 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E' 2026-03-20 16:26:25.849366 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-20 16:26:25.849371 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J' 2026-03-20 16:26:25.849375 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU' 2026-03-20 16:26:25.849379 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq' 2026-03-20 16:26:25.849383 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6' 2026-03-20 16:26:25.849387 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3' 2026-03-20 16:26:25.849393 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk' 2026-03-20 16:26:25.849398 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb' 2026-03-20 16:26:25.849402 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE' 2026-03-20 16:26:25.849406 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==' 2026-03-20 16:26:25.849412 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.849416 | controller | CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.849421 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw' 2026-03-20 16:26:25.849425 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx' 2026-03-20 16:26:25.849429 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb' 2026-03-20 16:26:25.849433 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2' 2026-03-20 16:26:25.849437 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro' 2026-03-20 16:26:25.849442 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671' 2026-03-20 16:26:25.849446 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B' 2026-03-20 16:26:25.849450 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ' 2026-03-20 16:26:25.849460 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-20 16:26:25.849465 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E' 2026-03-20 16:26:25.849469 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-20 16:26:25.849473 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J' 2026-03-20 16:26:25.849477 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU' 2026-03-20 16:26:25.849481 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq' 2026-03-20 16:26:25.849486 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6' 2026-03-20 16:26:25.849490 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3' 2026-03-20 16:26:25.849494 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk' 2026-03-20 16:26:25.849498 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb' 2026-03-20 16:26:25.849504 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE' 2026-03-20 16:26:25.849508 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==' 2026-03-20 16:26:25.849515 | controller | - ' logger.go:42: 16:19:06 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.849519 | controller | CERTIFICATE-----'' '']''' 2026-03-20 16:26:25.849523 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | running 2026-03-20 16:26:25.849527 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.849532 | controller | - ' public_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-20 16:26:25.849536 | controller | cat /etc/pki/tls/certs/public.crt)' 2026-03-20 16:26:25.849540 | controller | - ' public_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc 2026-03-20 16:26:25.849544 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-20 16:26:25.849549 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.849553 | controller | - ' if [ "${public_svc_cert}" != "${public_secret_cert}" ]; then' 2026-03-20 16:26:25.849557 | controller | - ' exit 1' 2026-03-20 16:26:25.849561 | controller | - ' fi' 2026-03-20 16:26:25.849565 | controller | - ' ' 2026-03-20 16:26:25.849570 | controller | - ' internal_svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 2026-03-20 16:26:25.849574 | controller | cat /etc/pki/tls/certs/internal.crt)' 2026-03-20 16:26:25.849578 | controller | - ' internal_secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc 2026-03-20 16:26:25.849582 | controller | -o jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-20 16:26:25.849586 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.849591 | controller | - ' if [ "${internal_svc_cert}" != "${internal_secret_cert}" ]; then' 2026-03-20 16:26:25.849595 | controller | - ' exit 1' 2026-03-20 16:26:25.849599 | controller | - ' fi' 2026-03-20 16:26:25.849603 | controller | - ' ]' 2026-03-20 16:26:25.849607 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-20 16:26:25.849612 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-20 16:26:25.849616 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | + public_svc_cert=''-----BEGIN 2026-03-20 16:26:25.849620 | controller | CERTIFICATE-----' 2026-03-20 16:26:25.849624 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY' 2026-03-20 16:26:25.849628 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx' 2026-03-20 16:26:25.849633 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu' 2026-03-20 16:26:25.849637 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH' 2026-03-20 16:26:25.849641 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol' 2026-03-20 16:26:25.849645 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X' 2026-03-20 16:26:25.849656 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI' 2026-03-20 16:26:25.849660 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28' 2026-03-20 16:26:25.849666 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww' 2026-03-20 16:26:25.849671 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/' 2026-03-20 16:26:25.849675 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy' 2026-03-20 16:26:25.849679 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH' 2026-03-20 16:26:25.849683 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp' 2026-03-20 16:26:25.849688 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6' 2026-03-20 16:26:25.849702 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv' 2026-03-20 16:26:25.849709 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl' 2026-03-20 16:26:25.849713 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs' 2026-03-20 16:26:25.849717 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0' 2026-03-20 16:26:25.849721 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9' 2026-03-20 16:26:25.849725 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=' 2026-03-20 16:26:25.849732 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.849736 | controller | CERTIFICATE-----''' 2026-03-20 16:26:25.849740 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-20 16:26:25.849745 | controller | get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}''' 2026-03-20 16:26:25.849749 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 2026-03-20 16:26:25.849753 | controller | --decode' 2026-03-20 16:26:25.849757 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | + public_secret_cert=''-----BEGIN 2026-03-20 16:26:25.849761 | controller | CERTIFICATE-----' 2026-03-20 16:26:25.849766 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY' 2026-03-20 16:26:25.849770 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx' 2026-03-20 16:26:25.849774 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu' 2026-03-20 16:26:25.849778 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH' 2026-03-20 16:26:25.849782 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol' 2026-03-20 16:26:25.849787 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X' 2026-03-20 16:26:25.849791 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI' 2026-03-20 16:26:25.849797 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28' 2026-03-20 16:26:25.849801 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww' 2026-03-20 16:26:25.849805 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/' 2026-03-20 16:26:25.849816 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy' 2026-03-20 16:26:25.849821 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH' 2026-03-20 16:26:25.849825 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp' 2026-03-20 16:26:25.849833 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6' 2026-03-20 16:26:25.849837 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv' 2026-03-20 16:26:25.849841 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl' 2026-03-20 16:26:25.849845 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs' 2026-03-20 16:26:25.849850 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0' 2026-03-20 16:26:25.849854 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9' 2026-03-20 16:26:25.849858 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=' 2026-03-20 16:26:25.849864 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.849869 | controller | CERTIFICATE-----''' 2026-03-20 16:26:25.849873 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' 2026-03-20 16:26:25.849877 | controller | ''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.849881 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY' 2026-03-20 16:26:25.849885 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx' 2026-03-20 16:26:25.849890 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu' 2026-03-20 16:26:25.849894 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH' 2026-03-20 16:26:25.849898 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol' 2026-03-20 16:26:25.849902 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X' 2026-03-20 16:26:25.849906 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI' 2026-03-20 16:26:25.849912 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28' 2026-03-20 16:26:25.849918 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww' 2026-03-20 16:26:25.849922 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/' 2026-03-20 16:26:25.849927 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy' 2026-03-20 16:26:25.849931 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH' 2026-03-20 16:26:25.849935 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp' 2026-03-20 16:26:25.849939 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6' 2026-03-20 16:26:25.849943 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv' 2026-03-20 16:26:25.849948 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl' 2026-03-20 16:26:25.849958 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs' 2026-03-20 16:26:25.849962 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0' 2026-03-20 16:26:25.849967 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9' 2026-03-20 16:26:25.849971 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=' 2026-03-20 16:26:25.849977 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.849982 | controller | CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.849986 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY' 2026-03-20 16:26:25.849990 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx' 2026-03-20 16:26:25.849994 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu' 2026-03-20 16:26:25.849998 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH' 2026-03-20 16:26:25.850003 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol' 2026-03-20 16:26:25.850007 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X' 2026-03-20 16:26:25.850011 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI' 2026-03-20 16:26:25.850015 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28' 2026-03-20 16:26:25.850019 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww' 2026-03-20 16:26:25.850023 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/' 2026-03-20 16:26:25.850029 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy' 2026-03-20 16:26:25.850034 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH' 2026-03-20 16:26:25.850038 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp' 2026-03-20 16:26:25.850042 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6' 2026-03-20 16:26:25.850046 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv' 2026-03-20 16:26:25.850050 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl' 2026-03-20 16:26:25.850054 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs' 2026-03-20 16:26:25.850059 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0' 2026-03-20 16:26:25.850063 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9' 2026-03-20 16:26:25.850067 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=' 2026-03-20 16:26:25.850073 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.850077 | controller | CERTIFICATE-----'' '']''' 2026-03-20 16:26:25.850081 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-20 16:26:25.850086 | controller | rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt' 2026-03-20 16:26:25.850090 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_svc_cert=''-----BEGIN 2026-03-20 16:26:25.850100 | controller | CERTIFICATE-----' 2026-03-20 16:26:25.850104 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw' 2026-03-20 16:26:25.850109 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx' 2026-03-20 16:26:25.850113 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb' 2026-03-20 16:26:25.850117 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2' 2026-03-20 16:26:25.850121 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro' 2026-03-20 16:26:25.850125 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671' 2026-03-20 16:26:25.850130 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B' 2026-03-20 16:26:25.850134 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ' 2026-03-20 16:26:25.850138 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-20 16:26:25.850142 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E' 2026-03-20 16:26:25.850149 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-20 16:26:25.850153 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J' 2026-03-20 16:26:25.850157 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU' 2026-03-20 16:26:25.850161 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq' 2026-03-20 16:26:25.850165 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6' 2026-03-20 16:26:25.850170 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3' 2026-03-20 16:26:25.850174 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk' 2026-03-20 16:26:25.850178 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb' 2026-03-20 16:26:25.850182 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE' 2026-03-20 16:26:25.850186 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==' 2026-03-20 16:26:25.850193 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.850197 | controller | CERTIFICATE-----''' 2026-03-20 16:26:25.850201 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | ++ oc 2026-03-20 16:26:25.850205 | controller | get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}''' 2026-03-20 16:26:25.850209 | controller | - ' logger.go:42: 16:19:07 | watcher-tls-certs-change/1-deploy-with-tlse | ++ base64 2026-03-20 16:26:25.850214 | controller | --decode' 2026-03-20 16:26:25.850218 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | + internal_secret_cert=''-----BEGIN 2026-03-20 16:26:25.850222 | controller | CERTIFICATE-----' 2026-03-20 16:26:25.850226 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw' 2026-03-20 16:26:25.850230 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx' 2026-03-20 16:26:25.850235 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb' 2026-03-20 16:26:25.850244 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2' 2026-03-20 16:26:25.850249 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro' 2026-03-20 16:26:25.850253 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671' 2026-03-20 16:26:25.850257 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B' 2026-03-20 16:26:25.850262 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ' 2026-03-20 16:26:25.850267 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-20 16:26:25.850272 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E' 2026-03-20 16:26:25.850276 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-20 16:26:25.850280 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J' 2026-03-20 16:26:25.850284 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU' 2026-03-20 16:26:25.850288 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq' 2026-03-20 16:26:25.850292 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6' 2026-03-20 16:26:25.850297 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3' 2026-03-20 16:26:25.850301 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk' 2026-03-20 16:26:25.850305 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb' 2026-03-20 16:26:25.850309 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE' 2026-03-20 16:26:25.850313 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==' 2026-03-20 16:26:25.850319 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.850324 | controller | CERTIFICATE-----''' 2026-03-20 16:26:25.850328 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | + ''['' 2026-03-20 16:26:25.850332 | controller | ''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.850336 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw' 2026-03-20 16:26:25.850341 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx' 2026-03-20 16:26:25.850345 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb' 2026-03-20 16:26:25.850349 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2' 2026-03-20 16:26:25.850353 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro' 2026-03-20 16:26:25.850357 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671' 2026-03-20 16:26:25.850362 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B' 2026-03-20 16:26:25.850366 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ' 2026-03-20 16:26:25.850370 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-20 16:26:25.850382 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E' 2026-03-20 16:26:25.850387 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-20 16:26:25.850391 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J' 2026-03-20 16:26:25.850395 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU' 2026-03-20 16:26:25.850399 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq' 2026-03-20 16:26:25.850404 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6' 2026-03-20 16:26:25.850408 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3' 2026-03-20 16:26:25.850412 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk' 2026-03-20 16:26:25.850416 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb' 2026-03-20 16:26:25.850420 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE' 2026-03-20 16:26:25.850425 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==' 2026-03-20 16:26:25.850431 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.850435 | controller | CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.850439 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw' 2026-03-20 16:26:25.850444 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx' 2026-03-20 16:26:25.850448 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb' 2026-03-20 16:26:25.850452 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2' 2026-03-20 16:26:25.850456 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro' 2026-03-20 16:26:25.850460 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671' 2026-03-20 16:26:25.850464 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B' 2026-03-20 16:26:25.850469 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ' 2026-03-20 16:26:25.850473 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-20 16:26:25.850477 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E' 2026-03-20 16:26:25.850481 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-20 16:26:25.850487 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J' 2026-03-20 16:26:25.850492 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU' 2026-03-20 16:26:25.850496 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq' 2026-03-20 16:26:25.850500 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6' 2026-03-20 16:26:25.850504 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3' 2026-03-20 16:26:25.850515 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk' 2026-03-20 16:26:25.850519 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb' 2026-03-20 16:26:25.850523 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE' 2026-03-20 16:26:25.850528 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==' 2026-03-20 16:26:25.850534 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | -----END 2026-03-20 16:26:25.850538 | controller | CERTIFICATE-----'' '']''' 2026-03-20 16:26:25.850542 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/1-deploy-with-tlse | test 2026-03-20 16:26:25.850547 | controller | step completed 1-deploy-with-tlse' 2026-03-20 16:26:25.850551 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850555 | controller | | starting test step 2-change-public-svc-certificate' 2026-03-20 16:26:25.850559 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850563 | controller | | running command: [sh -c oc patch Certificate -n $NAMESPACE watcher-public-svc 2026-03-20 16:26:25.850568 | controller | --type=''json'' -p=''[{"op": "replace", "path": "/spec/dnsNames", "value":[''watcher-public.watcher-kuttl-default.svc'', 2026-03-20 16:26:25.850572 | controller | ''watcher-public.watcher-kuttl-default.svc.cluster.local'']}]''' 2026-03-20 16:26:25.850576 | controller | - ' ]' 2026-03-20 16:26:25.850580 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850584 | controller | | certificate.cert-manager.io/watcher-public-svc patched' 2026-03-20 16:26:25.850588 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850593 | controller | | running command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.850597 | controller | - ' svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat 2026-03-20 16:26:25.850601 | controller | /etc/pki/tls/certs/public.crt)' 2026-03-20 16:26:25.850605 | controller | - ' secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath=''{.data.tls\.crt}'' 2026-03-20 16:26:25.850609 | controller | | base64 --decode)' 2026-03-20 16:26:25.850614 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.850618 | controller | - ' if [ "${svc_cert}" != "${secret_cert}" ]; then' 2026-03-20 16:26:25.850622 | controller | - ' exit 1' 2026-03-20 16:26:25.850626 | controller | - ' fi' 2026-03-20 16:26:25.850630 | controller | - ' ]' 2026-03-20 16:26:25.850634 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850639 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-20 16:26:25.850646 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850650 | controller | | + svc_cert=''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.850654 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850659 | controller | | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY' 2026-03-20 16:26:25.850663 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850667 | controller | | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx' 2026-03-20 16:26:25.850671 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850675 | controller | | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu' 2026-03-20 16:26:25.850679 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850684 | controller | | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH' 2026-03-20 16:26:25.850688 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850705 | controller | | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol' 2026-03-20 16:26:25.850711 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850715 | controller | | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X' 2026-03-20 16:26:25.850719 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850724 | controller | | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI' 2026-03-20 16:26:25.850728 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850732 | controller | | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28' 2026-03-20 16:26:25.850736 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850740 | controller | | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww' 2026-03-20 16:26:25.850744 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850749 | controller | | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/' 2026-03-20 16:26:25.850760 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850765 | controller | | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy' 2026-03-20 16:26:25.850769 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850773 | controller | | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH' 2026-03-20 16:26:25.850777 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850781 | controller | | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp' 2026-03-20 16:26:25.850786 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850790 | controller | | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6' 2026-03-20 16:26:25.850794 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850798 | controller | | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv' 2026-03-20 16:26:25.850802 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850806 | controller | | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl' 2026-03-20 16:26:25.850810 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850815 | controller | | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs' 2026-03-20 16:26:25.850819 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850825 | controller | | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0' 2026-03-20 16:26:25.850829 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850834 | controller | | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9' 2026-03-20 16:26:25.850838 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850842 | controller | | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=' 2026-03-20 16:26:25.850846 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850850 | controller | | -----END CERTIFICATE-----''' 2026-03-20 16:26:25.850855 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850859 | controller | | ++ base64 --decode' 2026-03-20 16:26:25.850863 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850867 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}''' 2026-03-20 16:26:25.850871 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850875 | controller | | + secret_cert=''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.850880 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850884 | controller | | MIID6jCCAlKgAwIBAgIQZJaVw8Wo/ookNMvl5tSf7zANBgkqhkiG9w0BAQwFADAY' 2026-03-20 16:26:25.850888 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850892 | controller | | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTkwOFoXDTMxMDMx' 2026-03-20 16:26:25.850896 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850902 | controller | | OTE2MTkwOFowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKXRB7B3' 2026-03-20 16:26:25.850907 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850911 | controller | | rhfYNMtE5MNaVBPTQ1ChoGf12bhDR7Hfz4Pv0e9XQRR0iY1L3O/sIYBsUs+UDZIi' 2026-03-20 16:26:25.850915 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850919 | controller | | 1sO1e0hu0f/ub72rJMdYIWEW+gWSTmTOFffkt+dnobBgQlXXw5g7ZSg4EbaGx6Kf' 2026-03-20 16:26:25.850923 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850927 | controller | | GKGVlEhQ9uEjQNrifD33D8vfR/3AsqR1MnZEcrQRXHjjhDXlhLNtNQEceenSDm+0' 2026-03-20 16:26:25.850932 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850936 | controller | | 0ok9qcjT0JDZCVre77nyPUISheJeWK3ZkiHxDthkL7ukO/9rVwYRNuWEB4JHZHFI' 2026-03-20 16:26:25.850940 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850944 | controller | | MqGuhbe7jgVW0QrBl314Zjw5ET2FKdlTKtd/b+PhNnuU2qhjWYO76WhqcpllFwau' 2026-03-20 16:26:25.850948 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850952 | controller | | sunqw0flNMujB88CAwEAAaOBxzCBxDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww' 2026-03-20 16:26:25.850956 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850960 | controller | | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/' 2026-03-20 16:26:25.850965 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850969 | controller | | 0Hm6PaQEE6ByeDBuBgNVHREBAf8EZDBigih3YXRjaGVyLXB1YmxpYy53YXRjaGVy' 2026-03-20 16:26:25.850980 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850984 | controller | | LWt1dHRsLWRlZmF1bHQuc3ZjgjZ3YXRjaGVyLXB1YmxpYy53YXRjaGVyLWt1dHRs' 2026-03-20 16:26:25.850991 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.850995 | controller | | LWRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwwDQYJKoZIhvcNAQEMBQADggGBAC/o' 2026-03-20 16:26:25.850999 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851003 | controller | | 5/i5C0Y/AKLiB3xtslI8c9UVyiOwIpEOrBpAYh0CQe7PkZ9zV4mBLHrY5tneZeQ/' 2026-03-20 16:26:25.851009 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851013 | controller | | gKY2iSGZifksOo9R2RmPjpKPIrDqgKveIqRYsNeCkVKKXimGwzNVBhoCDnpkrd0r' 2026-03-20 16:26:25.851018 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851022 | controller | | HttXDFUG47HTtNbtx9gbHEnL9JViBLG/91Fb2PavY1G6rYeRpJT13dAr2/ozwk0F' 2026-03-20 16:26:25.851026 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851030 | controller | | j5lPZ2ZxWmWxE51A3yAmXXa8NT+MJpwWW1Gu9eSb+Bn2TELEMuEHos6Qn6Ob957P' 2026-03-20 16:26:25.851034 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851038 | controller | | YcG0oFRp+ecTVjfLhZmXMyYoEzmLlXLuGaUQqvlXECGTSbw5erNtKRkS4gAQ0PAI' 2026-03-20 16:26:25.851043 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851047 | controller | | sRSypaVKNLM96+09mRoN9ppbVXUrmd4haOg9l09Bpth6s6bywa3GcqkBHjORfRUu' 2026-03-20 16:26:25.851051 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851055 | controller | | 6/lzDFbw+LAZRB7kiWeyD9W+LlhYC6ZzxjrTpuNIfNwmsCKCPzHadwq2OXa2vIZi' 2026-03-20 16:26:25.851059 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851063 | controller | | p+EfRtIGSWnMfKbcf6w2QsDS8061IT7HHoFOS9K8BWVPA3gUgAn4f0+B5r5t1g==' 2026-03-20 16:26:25.851068 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851072 | controller | | -----END CERTIFICATE-----''' 2026-03-20 16:26:25.851076 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851080 | controller | | + ''['' ''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.851084 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851088 | controller | | MIIDsjCCAhqgAwIBAgIQQGgnPKJx6caSondqrX1YlDANBgkqhkiG9w0BAQwFADAY' 2026-03-20 16:26:25.851093 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851097 | controller | | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTg0M1oXDTMxMDMx' 2026-03-20 16:26:25.851101 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851105 | controller | | OTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALEskHzu' 2026-03-20 16:26:25.851109 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851113 | controller | | pLI41onIaipoKjX9plnVLg+kUeNGTISsrlyrtTpfPbOsXl0kz19XhwZGYQbwSACH' 2026-03-20 16:26:25.851118 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851122 | controller | | UWE17x6pE3Q4lQFoTxSzPhW82AWqxJcJ5kC0EzfCr16G+Qtb11aG7Xq26LFjVsol' 2026-03-20 16:26:25.851126 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851130 | controller | | xXlj9jVJbUl9l/T6FvywxM8rNMCmfYcTlmZX6B1EPy8uqWt2p5g8o4i9H7q8xq7X' 2026-03-20 16:26:25.851134 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851138 | controller | | b0P3Oh9vC5aRtE061l7oel8MakPaerm/9724fDamL/bu0uw1nIq/D7gsiHCbB1iI' 2026-03-20 16:26:25.851142 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851148 | controller | | j9FfbDP+zFqZjiO6Kbti5vBiLiBOuCrHD1AGqmbPaUP9GBSCwgflBbzdY6fYkt28' 2026-03-20 16:26:25.851153 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851157 | controller | | w4ZKy8v3otPqXwMCAwEAAaOBjzCBjDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww' 2026-03-20 16:26:25.851161 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851165 | controller | | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/' 2026-03-20 16:26:25.851169 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851173 | controller | | 0Hm6PaQEE6ByeDA2BgNVHREBAf8ELDAqgih3YXRjaGVyLXB1YmxpYy53YXRjaGVy' 2026-03-20 16:26:25.851178 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851182 | controller | | LWt1dHRsLWRlZmF1bHQuc3ZjMA0GCSqGSIb3DQEBDAUAA4IBgQCYdGi/BrpyMcuH' 2026-03-20 16:26:25.851186 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851190 | controller | | oMuGGt9nN+6i/oVF2dXgtd8e76oZbeHeAbwDNLGWm0bqaBfQvv+tMOR7V+9Akdvp' 2026-03-20 16:26:25.851201 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851206 | controller | | 6hBgnyeXMwEZuPJyRWfNtYTjEtg/3t0TWoBbu7qSb+++p4DDGgMmW5XaH3roSOj6' 2026-03-20 16:26:25.851210 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851214 | controller | | 3WXJ18Hz3kP/zI96JL+a1P7WsPRCULsc7wCnU9llvj9k8pI+vgZFaP9D8oFOTZAv' 2026-03-20 16:26:25.851218 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851223 | controller | | vL9vbj0AA4GlN3pkx000XenYwddqiIFQ9YqivvZ8VH7DHHYiTvLucnKYUStJY7Wl' 2026-03-20 16:26:25.851227 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851231 | controller | | 7ttC6OLUkQd4Ncf7GyAXON3qdOAKzC3Hs0DHYv4NAicb+WuuVjWr+Sgb3O3/WTxs' 2026-03-20 16:26:25.851235 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851241 | controller | | R0t88gl20NLzpZY6+Fbt14oNjDcr5D6VN39PXTa72a5NpEVEEw9+OVcprBHRSmj0' 2026-03-20 16:26:25.851246 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851250 | controller | | ZdDPFpNgfutSvFqAOSIU/pLjwzCAa30OwiaDFtUZCMFLjivDqVU/7DxMyAixbmy9' 2026-03-20 16:26:25.851254 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851258 | controller | | dfxEbZsJCEdVZBOu9H56zByyY9A7FGzavHi3NrpLh6B1eaB3SrI=' 2026-03-20 16:26:25.851262 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851266 | controller | | -----END CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.851271 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851275 | controller | | MIID6jCCAlKgAwIBAgIQZJaVw8Wo/ookNMvl5tSf7zANBgkqhkiG9w0BAQwFADAY' 2026-03-20 16:26:25.851281 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851285 | controller | | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTkwOFoXDTMxMDMx' 2026-03-20 16:26:25.851289 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851293 | controller | | OTE2MTkwOFowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKXRB7B3' 2026-03-20 16:26:25.851297 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851301 | controller | | rhfYNMtE5MNaVBPTQ1ChoGf12bhDR7Hfz4Pv0e9XQRR0iY1L3O/sIYBsUs+UDZIi' 2026-03-20 16:26:25.851305 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851310 | controller | | 1sO1e0hu0f/ub72rJMdYIWEW+gWSTmTOFffkt+dnobBgQlXXw5g7ZSg4EbaGx6Kf' 2026-03-20 16:26:25.851315 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851320 | controller | | GKGVlEhQ9uEjQNrifD33D8vfR/3AsqR1MnZEcrQRXHjjhDXlhLNtNQEceenSDm+0' 2026-03-20 16:26:25.851324 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851328 | controller | | 0ok9qcjT0JDZCVre77nyPUISheJeWK3ZkiHxDthkL7ukO/9rVwYRNuWEB4JHZHFI' 2026-03-20 16:26:25.851332 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851336 | controller | | MqGuhbe7jgVW0QrBl314Zjw5ET2FKdlTKtd/b+PhNnuU2qhjWYO76WhqcpllFwau' 2026-03-20 16:26:25.851340 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851345 | controller | | sunqw0flNMujB88CAwEAAaOBxzCBxDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww' 2026-03-20 16:26:25.851349 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851353 | controller | | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/' 2026-03-20 16:26:25.851357 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851361 | controller | | 0Hm6PaQEE6ByeDBuBgNVHREBAf8EZDBigih3YXRjaGVyLXB1YmxpYy53YXRjaGVy' 2026-03-20 16:26:25.851366 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851370 | controller | | LWt1dHRsLWRlZmF1bHQuc3ZjgjZ3YXRjaGVyLXB1YmxpYy53YXRjaGVyLWt1dHRs' 2026-03-20 16:26:25.851374 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851378 | controller | | LWRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwwDQYJKoZIhvcNAQEMBQADggGBAC/o' 2026-03-20 16:26:25.851382 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851386 | controller | | 5/i5C0Y/AKLiB3xtslI8c9UVyiOwIpEOrBpAYh0CQe7PkZ9zV4mBLHrY5tneZeQ/' 2026-03-20 16:26:25.851390 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851395 | controller | | gKY2iSGZifksOo9R2RmPjpKPIrDqgKveIqRYsNeCkVKKXimGwzNVBhoCDnpkrd0r' 2026-03-20 16:26:25.851399 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851403 | controller | | HttXDFUG47HTtNbtx9gbHEnL9JViBLG/91Fb2PavY1G6rYeRpJT13dAr2/ozwk0F' 2026-03-20 16:26:25.851407 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851577 | controller | | j5lPZ2ZxWmWxE51A3yAmXXa8NT+MJpwWW1Gu9eSb+Bn2TELEMuEHos6Qn6Ob957P' 2026-03-20 16:26:25.851582 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851586 | controller | | YcG0oFRp+ecTVjfLhZmXMyYoEzmLlXLuGaUQqvlXECGTSbw5erNtKRkS4gAQ0PAI' 2026-03-20 16:26:25.851590 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851594 | controller | | sRSypaVKNLM96+09mRoN9ppbVXUrmd4haOg9l09Bpth6s6bywa3GcqkBHjORfRUu' 2026-03-20 16:26:25.851599 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851603 | controller | | 6/lzDFbw+LAZRB7kiWeyD9W+LlhYC6ZzxjrTpuNIfNwmsCKCPzHadwq2OXa2vIZi' 2026-03-20 16:26:25.851607 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851611 | controller | | p+EfRtIGSWnMfKbcf6w2QsDS8061IT7HHoFOS9K8BWVPA3gUgAn4f0+B5r5t1g==' 2026-03-20 16:26:25.851615 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851619 | controller | | -----END CERTIFICATE-----'' '']''' 2026-03-20 16:26:25.851624 | controller | - ' logger.go:42: 16:19:08 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851628 | controller | | + exit 1' 2026-03-20 16:26:25.851632 | controller | - ' logger.go:42: 16:19:09 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851638 | controller | | running command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.851643 | controller | - ' svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat 2026-03-20 16:26:25.851647 | controller | /etc/pki/tls/certs/public.crt)' 2026-03-20 16:26:25.851653 | controller | - ' secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath=''{.data.tls\.crt}'' 2026-03-20 16:26:25.851658 | controller | | base64 --decode)' 2026-03-20 16:26:25.851662 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.851666 | controller | - ' if [ "${svc_cert}" != "${secret_cert}" ]; then' 2026-03-20 16:26:25.851671 | controller | - ' exit 1' 2026-03-20 16:26:25.851675 | controller | - ' fi' 2026-03-20 16:26:25.851679 | controller | - ' ]' 2026-03-20 16:26:25.851683 | controller | - ' logger.go:42: 16:19:09 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851688 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-20 16:26:25.851702 | controller | - ' logger.go:42: 16:19:09 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851708 | controller | | error: unable to upgrade connection: container not found ("watcher-api")' 2026-03-20 16:26:25.851713 | controller | - ' logger.go:42: 16:19:09 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851717 | controller | | + svc_cert=' 2026-03-20 16:26:25.851721 | controller | - ' logger.go:42: 16:19:10 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851725 | controller | | running command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.851730 | controller | - ' svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat 2026-03-20 16:26:25.851734 | controller | /etc/pki/tls/certs/public.crt)' 2026-03-20 16:26:25.851738 | controller | - ' secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath=''{.data.tls\.crt}'' 2026-03-20 16:26:25.851742 | controller | | base64 --decode)' 2026-03-20 16:26:25.851747 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.851751 | controller | - ' if [ "${svc_cert}" != "${secret_cert}" ]; then' 2026-03-20 16:26:25.851755 | controller | - ' exit 1' 2026-03-20 16:26:25.851759 | controller | - ' fi' 2026-03-20 16:26:25.851764 | controller | - ' ]' 2026-03-20 16:26:25.851785 | controller | - ' logger.go:42: 16:19:10 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851789 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-20 16:26:25.851796 | controller | - ' logger.go:42: 16:19:10 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851800 | controller | | error: unable to upgrade connection: container not found ("watcher-api")' 2026-03-20 16:26:25.851805 | controller | - ' logger.go:42: 16:19:10 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851809 | controller | | + svc_cert=' 2026-03-20 16:26:25.851815 | controller | - ' logger.go:42: 16:19:11 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851819 | controller | | running command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.851823 | controller | - ' svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat 2026-03-20 16:26:25.851827 | controller | /etc/pki/tls/certs/public.crt)' 2026-03-20 16:26:25.851832 | controller | - ' secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-public-svc -o jsonpath=''{.data.tls\.crt}'' 2026-03-20 16:26:25.851836 | controller | | base64 --decode)' 2026-03-20 16:26:25.851840 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.851845 | controller | - ' if [ "${svc_cert}" != "${secret_cert}" ]; then' 2026-03-20 16:26:25.851849 | controller | - ' exit 1' 2026-03-20 16:26:25.851855 | controller | - ' fi' 2026-03-20 16:26:25.851859 | controller | - ' ]' 2026-03-20 16:26:25.851863 | controller | - ' logger.go:42: 16:19:11 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851870 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/public.crt' 2026-03-20 16:26:25.851874 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851886 | controller | | + svc_cert=''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.851891 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851898 | controller | | MIID6jCCAlKgAwIBAgIQZJaVw8Wo/ookNMvl5tSf7zANBgkqhkiG9w0BAQwFADAY' 2026-03-20 16:26:25.851903 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851907 | controller | | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTkwOFoXDTMxMDMx' 2026-03-20 16:26:25.851911 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851915 | controller | | OTE2MTkwOFowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKXRB7B3' 2026-03-20 16:26:25.851920 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851924 | controller | | rhfYNMtE5MNaVBPTQ1ChoGf12bhDR7Hfz4Pv0e9XQRR0iY1L3O/sIYBsUs+UDZIi' 2026-03-20 16:26:25.851928 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851932 | controller | | 1sO1e0hu0f/ub72rJMdYIWEW+gWSTmTOFffkt+dnobBgQlXXw5g7ZSg4EbaGx6Kf' 2026-03-20 16:26:25.851936 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851941 | controller | | GKGVlEhQ9uEjQNrifD33D8vfR/3AsqR1MnZEcrQRXHjjhDXlhLNtNQEceenSDm+0' 2026-03-20 16:26:25.851945 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851949 | controller | | 0ok9qcjT0JDZCVre77nyPUISheJeWK3ZkiHxDthkL7ukO/9rVwYRNuWEB4JHZHFI' 2026-03-20 16:26:25.851953 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851957 | controller | | MqGuhbe7jgVW0QrBl314Zjw5ET2FKdlTKtd/b+PhNnuU2qhjWYO76WhqcpllFwau' 2026-03-20 16:26:25.851962 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851966 | controller | | sunqw0flNMujB88CAwEAAaOBxzCBxDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww' 2026-03-20 16:26:25.851970 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851974 | controller | | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/' 2026-03-20 16:26:25.851978 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851984 | controller | | 0Hm6PaQEE6ByeDBuBgNVHREBAf8EZDBigih3YXRjaGVyLXB1YmxpYy53YXRjaGVy' 2026-03-20 16:26:25.851989 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.851993 | controller | | LWt1dHRsLWRlZmF1bHQuc3ZjgjZ3YXRjaGVyLXB1YmxpYy53YXRjaGVyLWt1dHRs' 2026-03-20 16:26:25.851997 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852001 | controller | | LWRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwwDQYJKoZIhvcNAQEMBQADggGBAC/o' 2026-03-20 16:26:25.852005 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852010 | controller | | 5/i5C0Y/AKLiB3xtslI8c9UVyiOwIpEOrBpAYh0CQe7PkZ9zV4mBLHrY5tneZeQ/' 2026-03-20 16:26:25.852014 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852018 | controller | | gKY2iSGZifksOo9R2RmPjpKPIrDqgKveIqRYsNeCkVKKXimGwzNVBhoCDnpkrd0r' 2026-03-20 16:26:25.852022 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852026 | controller | | HttXDFUG47HTtNbtx9gbHEnL9JViBLG/91Fb2PavY1G6rYeRpJT13dAr2/ozwk0F' 2026-03-20 16:26:25.852031 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852037 | controller | | j5lPZ2ZxWmWxE51A3yAmXXa8NT+MJpwWW1Gu9eSb+Bn2TELEMuEHos6Qn6Ob957P' 2026-03-20 16:26:25.852041 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852045 | controller | | YcG0oFRp+ecTVjfLhZmXMyYoEzmLlXLuGaUQqvlXECGTSbw5erNtKRkS4gAQ0PAI' 2026-03-20 16:26:25.852049 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852054 | controller | | sRSypaVKNLM96+09mRoN9ppbVXUrmd4haOg9l09Bpth6s6bywa3GcqkBHjORfRUu' 2026-03-20 16:26:25.852058 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852062 | controller | | 6/lzDFbw+LAZRB7kiWeyD9W+LlhYC6ZzxjrTpuNIfNwmsCKCPzHadwq2OXa2vIZi' 2026-03-20 16:26:25.852066 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852070 | controller | | p+EfRtIGSWnMfKbcf6w2QsDS8061IT7HHoFOS9K8BWVPA3gUgAn4f0+B5r5t1g==' 2026-03-20 16:26:25.852075 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852079 | controller | | -----END CERTIFICATE-----''' 2026-03-20 16:26:25.852083 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852087 | controller | | ++ base64 --decode' 2026-03-20 16:26:25.852091 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852096 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-public-svc -o ''jsonpath={.data.tls\.crt}''' 2026-03-20 16:26:25.852100 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852113 | controller | | + secret_cert=''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.852117 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852121 | controller | | MIID6jCCAlKgAwIBAgIQZJaVw8Wo/ookNMvl5tSf7zANBgkqhkiG9w0BAQwFADAY' 2026-03-20 16:26:25.852125 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852130 | controller | | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTkwOFoXDTMxMDMx' 2026-03-20 16:26:25.852134 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852138 | controller | | OTE2MTkwOFowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKXRB7B3' 2026-03-20 16:26:25.852142 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852146 | controller | | rhfYNMtE5MNaVBPTQ1ChoGf12bhDR7Hfz4Pv0e9XQRR0iY1L3O/sIYBsUs+UDZIi' 2026-03-20 16:26:25.852151 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852155 | controller | | 1sO1e0hu0f/ub72rJMdYIWEW+gWSTmTOFffkt+dnobBgQlXXw5g7ZSg4EbaGx6Kf' 2026-03-20 16:26:25.852159 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852163 | controller | | GKGVlEhQ9uEjQNrifD33D8vfR/3AsqR1MnZEcrQRXHjjhDXlhLNtNQEceenSDm+0' 2026-03-20 16:26:25.852167 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852171 | controller | | 0ok9qcjT0JDZCVre77nyPUISheJeWK3ZkiHxDthkL7ukO/9rVwYRNuWEB4JHZHFI' 2026-03-20 16:26:25.852176 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852180 | controller | | MqGuhbe7jgVW0QrBl314Zjw5ET2FKdlTKtd/b+PhNnuU2qhjWYO76WhqcpllFwau' 2026-03-20 16:26:25.852184 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852188 | controller | | sunqw0flNMujB88CAwEAAaOBxzCBxDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww' 2026-03-20 16:26:25.852192 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852197 | controller | | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/' 2026-03-20 16:26:25.852214 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852219 | controller | | 0Hm6PaQEE6ByeDBuBgNVHREBAf8EZDBigih3YXRjaGVyLXB1YmxpYy53YXRjaGVy' 2026-03-20 16:26:25.852223 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852227 | controller | | LWt1dHRsLWRlZmF1bHQuc3ZjgjZ3YXRjaGVyLXB1YmxpYy53YXRjaGVyLWt1dHRs' 2026-03-20 16:26:25.852231 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852235 | controller | | LWRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwwDQYJKoZIhvcNAQEMBQADggGBAC/o' 2026-03-20 16:26:25.852239 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852244 | controller | | 5/i5C0Y/AKLiB3xtslI8c9UVyiOwIpEOrBpAYh0CQe7PkZ9zV4mBLHrY5tneZeQ/' 2026-03-20 16:26:25.852248 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852252 | controller | | gKY2iSGZifksOo9R2RmPjpKPIrDqgKveIqRYsNeCkVKKXimGwzNVBhoCDnpkrd0r' 2026-03-20 16:26:25.852256 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852260 | controller | | HttXDFUG47HTtNbtx9gbHEnL9JViBLG/91Fb2PavY1G6rYeRpJT13dAr2/ozwk0F' 2026-03-20 16:26:25.852264 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852269 | controller | | j5lPZ2ZxWmWxE51A3yAmXXa8NT+MJpwWW1Gu9eSb+Bn2TELEMuEHos6Qn6Ob957P' 2026-03-20 16:26:25.852273 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852277 | controller | | YcG0oFRp+ecTVjfLhZmXMyYoEzmLlXLuGaUQqvlXECGTSbw5erNtKRkS4gAQ0PAI' 2026-03-20 16:26:25.852281 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852285 | controller | | sRSypaVKNLM96+09mRoN9ppbVXUrmd4haOg9l09Bpth6s6bywa3GcqkBHjORfRUu' 2026-03-20 16:26:25.852290 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852294 | controller | | 6/lzDFbw+LAZRB7kiWeyD9W+LlhYC6ZzxjrTpuNIfNwmsCKCPzHadwq2OXa2vIZi' 2026-03-20 16:26:25.852298 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852302 | controller | | p+EfRtIGSWnMfKbcf6w2QsDS8061IT7HHoFOS9K8BWVPA3gUgAn4f0+B5r5t1g==' 2026-03-20 16:26:25.852306 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852310 | controller | | -----END CERTIFICATE-----''' 2026-03-20 16:26:25.852315 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852319 | controller | | + ''['' ''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.852323 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852327 | controller | | MIID6jCCAlKgAwIBAgIQZJaVw8Wo/ookNMvl5tSf7zANBgkqhkiG9w0BAQwFADAY' 2026-03-20 16:26:25.852331 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852342 | controller | | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTkwOFoXDTMxMDMx' 2026-03-20 16:26:25.852347 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852351 | controller | | OTE2MTkwOFowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKXRB7B3' 2026-03-20 16:26:25.852355 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852360 | controller | | rhfYNMtE5MNaVBPTQ1ChoGf12bhDR7Hfz4Pv0e9XQRR0iY1L3O/sIYBsUs+UDZIi' 2026-03-20 16:26:25.852364 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852370 | controller | | 1sO1e0hu0f/ub72rJMdYIWEW+gWSTmTOFffkt+dnobBgQlXXw5g7ZSg4EbaGx6Kf' 2026-03-20 16:26:25.852375 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852380 | controller | | GKGVlEhQ9uEjQNrifD33D8vfR/3AsqR1MnZEcrQRXHjjhDXlhLNtNQEceenSDm+0' 2026-03-20 16:26:25.852385 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852389 | controller | | 0ok9qcjT0JDZCVre77nyPUISheJeWK3ZkiHxDthkL7ukO/9rVwYRNuWEB4JHZHFI' 2026-03-20 16:26:25.852393 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852399 | controller | | MqGuhbe7jgVW0QrBl314Zjw5ET2FKdlTKtd/b+PhNnuU2qhjWYO76WhqcpllFwau' 2026-03-20 16:26:25.852403 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852407 | controller | | sunqw0flNMujB88CAwEAAaOBxzCBxDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww' 2026-03-20 16:26:25.852412 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852416 | controller | | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/' 2026-03-20 16:26:25.852420 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852424 | controller | | 0Hm6PaQEE6ByeDBuBgNVHREBAf8EZDBigih3YXRjaGVyLXB1YmxpYy53YXRjaGVy' 2026-03-20 16:26:25.852430 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852434 | controller | | LWt1dHRsLWRlZmF1bHQuc3ZjgjZ3YXRjaGVyLXB1YmxpYy53YXRjaGVyLWt1dHRs' 2026-03-20 16:26:25.852438 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852443 | controller | | LWRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwwDQYJKoZIhvcNAQEMBQADggGBAC/o' 2026-03-20 16:26:25.852447 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852451 | controller | | 5/i5C0Y/AKLiB3xtslI8c9UVyiOwIpEOrBpAYh0CQe7PkZ9zV4mBLHrY5tneZeQ/' 2026-03-20 16:26:25.852455 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852459 | controller | | gKY2iSGZifksOo9R2RmPjpKPIrDqgKveIqRYsNeCkVKKXimGwzNVBhoCDnpkrd0r' 2026-03-20 16:26:25.852463 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852467 | controller | | HttXDFUG47HTtNbtx9gbHEnL9JViBLG/91Fb2PavY1G6rYeRpJT13dAr2/ozwk0F' 2026-03-20 16:26:25.852472 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852476 | controller | | j5lPZ2ZxWmWxE51A3yAmXXa8NT+MJpwWW1Gu9eSb+Bn2TELEMuEHos6Qn6Ob957P' 2026-03-20 16:26:25.852480 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852484 | controller | | YcG0oFRp+ecTVjfLhZmXMyYoEzmLlXLuGaUQqvlXECGTSbw5erNtKRkS4gAQ0PAI' 2026-03-20 16:26:25.852488 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852492 | controller | | sRSypaVKNLM96+09mRoN9ppbVXUrmd4haOg9l09Bpth6s6bywa3GcqkBHjORfRUu' 2026-03-20 16:26:25.852497 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852501 | controller | | 6/lzDFbw+LAZRB7kiWeyD9W+LlhYC6ZzxjrTpuNIfNwmsCKCPzHadwq2OXa2vIZi' 2026-03-20 16:26:25.852505 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852509 | controller | | p+EfRtIGSWnMfKbcf6w2QsDS8061IT7HHoFOS9K8BWVPA3gUgAn4f0+B5r5t1g==' 2026-03-20 16:26:25.852513 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852517 | controller | | -----END CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.852521 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852526 | controller | | MIID6jCCAlKgAwIBAgIQZJaVw8Wo/ookNMvl5tSf7zANBgkqhkiG9w0BAQwFADAY' 2026-03-20 16:26:25.852530 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852643 | controller | | MRYwFAYDVQQDEw1yb290Y2EtcHVibGljMB4XDTI2MDMyMDE2MTkwOFoXDTMxMDMx' 2026-03-20 16:26:25.852647 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852651 | controller | | OTE2MTkwOFowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKXRB7B3' 2026-03-20 16:26:25.852655 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852659 | controller | | rhfYNMtE5MNaVBPTQ1ChoGf12bhDR7Hfz4Pv0e9XQRR0iY1L3O/sIYBsUs+UDZIi' 2026-03-20 16:26:25.852671 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852675 | controller | | 1sO1e0hu0f/ub72rJMdYIWEW+gWSTmTOFffkt+dnobBgQlXXw5g7ZSg4EbaGx6Kf' 2026-03-20 16:26:25.852680 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852684 | controller | | GKGVlEhQ9uEjQNrifD33D8vfR/3AsqR1MnZEcrQRXHjjhDXlhLNtNQEceenSDm+0' 2026-03-20 16:26:25.852688 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852703 | controller | | 0ok9qcjT0JDZCVre77nyPUISheJeWK3ZkiHxDthkL7ukO/9rVwYRNuWEB4JHZHFI' 2026-03-20 16:26:25.852709 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852713 | controller | | MqGuhbe7jgVW0QrBl314Zjw5ET2FKdlTKtd/b+PhNnuU2qhjWYO76WhqcpllFwau' 2026-03-20 16:26:25.852717 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852722 | controller | | sunqw0flNMujB88CAwEAAaOBxzCBxDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww' 2026-03-20 16:26:25.852726 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852730 | controller | | CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBSxb24uCXYNsB7/' 2026-03-20 16:26:25.852734 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852738 | controller | | 0Hm6PaQEE6ByeDBuBgNVHREBAf8EZDBigih3YXRjaGVyLXB1YmxpYy53YXRjaGVy' 2026-03-20 16:26:25.852742 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852746 | controller | | LWt1dHRsLWRlZmF1bHQuc3ZjgjZ3YXRjaGVyLXB1YmxpYy53YXRjaGVyLWt1dHRs' 2026-03-20 16:26:25.852751 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852755 | controller | | LWRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwwDQYJKoZIhvcNAQEMBQADggGBAC/o' 2026-03-20 16:26:25.852759 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852763 | controller | | 5/i5C0Y/AKLiB3xtslI8c9UVyiOwIpEOrBpAYh0CQe7PkZ9zV4mBLHrY5tneZeQ/' 2026-03-20 16:26:25.852767 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852772 | controller | | gKY2iSGZifksOo9R2RmPjpKPIrDqgKveIqRYsNeCkVKKXimGwzNVBhoCDnpkrd0r' 2026-03-20 16:26:25.852776 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852780 | controller | | HttXDFUG47HTtNbtx9gbHEnL9JViBLG/91Fb2PavY1G6rYeRpJT13dAr2/ozwk0F' 2026-03-20 16:26:25.852784 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852790 | controller | | j5lPZ2ZxWmWxE51A3yAmXXa8NT+MJpwWW1Gu9eSb+Bn2TELEMuEHos6Qn6Ob957P' 2026-03-20 16:26:25.852795 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852799 | controller | | YcG0oFRp+ecTVjfLhZmXMyYoEzmLlXLuGaUQqvlXECGTSbw5erNtKRkS4gAQ0PAI' 2026-03-20 16:26:25.852803 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852809 | controller | | sRSypaVKNLM96+09mRoN9ppbVXUrmd4haOg9l09Bpth6s6bywa3GcqkBHjORfRUu' 2026-03-20 16:26:25.852813 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852817 | controller | | 6/lzDFbw+LAZRB7kiWeyD9W+LlhYC6ZzxjrTpuNIfNwmsCKCPzHadwq2OXa2vIZi' 2026-03-20 16:26:25.852824 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852828 | controller | | p+EfRtIGSWnMfKbcf6w2QsDS8061IT7HHoFOS9K8BWVPA3gUgAn4f0+B5r5t1g==' 2026-03-20 16:26:25.852832 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852836 | controller | | -----END CERTIFICATE-----'' '']''' 2026-03-20 16:26:25.852840 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/2-change-public-svc-certificate 2026-03-20 16:26:25.852845 | controller | | test step completed 2-change-public-svc-certificate' 2026-03-20 16:26:25.852849 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.852853 | controller | | starting test step 3-change-internal-svc-certificate' 2026-03-20 16:26:25.852857 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.852862 | controller | | running command: [sh -c oc patch Certificate -n $NAMESPACE watcher-internal-svc 2026-03-20 16:26:25.852866 | controller | --type=''json'' -p=''[{"op": "replace", "path": "/spec/dnsNames", "value":[''watcher-internal.watcher-kuttl-default.svc'', 2026-03-20 16:26:25.852870 | controller | ''watcher-internal.watcher-kuttl-default.svc.cluster.local'']}]''' 2026-03-20 16:26:25.852874 | controller | - ' ]' 2026-03-20 16:26:25.852878 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.852883 | controller | | certificate.cert-manager.io/watcher-internal-svc patched' 2026-03-20 16:26:25.852887 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.852891 | controller | | running command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.852895 | controller | - ' svc_cert=$(oc rsh -n $NAMESPACE -c watcher-api watcher-kuttl-api-0 cat 2026-03-20 16:26:25.852899 | controller | /etc/pki/tls/certs/internal.crt)' 2026-03-20 16:26:25.852911 | controller | - ' secret_cert=$(oc get -n $NAMESPACE secret cert-watcher-internal-svc -o 2026-03-20 16:26:25.852915 | controller | jsonpath=''{.data.tls\.crt}'' | base64 --decode)' 2026-03-20 16:26:25.852920 | controller | - ' # ensure that the svc secret and cert secret match' 2026-03-20 16:26:25.852924 | controller | - ' if [ "${svc_cert}" != "${secret_cert}" ]; then' 2026-03-20 16:26:25.852928 | controller | - ' exit 1' 2026-03-20 16:26:25.852932 | controller | - ' fi' 2026-03-20 16:26:25.852937 | controller | - ' ]' 2026-03-20 16:26:25.852941 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.852945 | controller | | ++ oc rsh -n watcher-kuttl-default -c watcher-api watcher-kuttl-api-0 cat /etc/pki/tls/certs/internal.crt' 2026-03-20 16:26:25.852949 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.852954 | controller | | + svc_cert=''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.852958 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.852962 | controller | | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw' 2026-03-20 16:26:25.852966 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.852970 | controller | | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx' 2026-03-20 16:26:25.852974 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.852978 | controller | | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb' 2026-03-20 16:26:25.852983 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.852987 | controller | | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2' 2026-03-20 16:26:25.852991 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.852997 | controller | | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro' 2026-03-20 16:26:25.853002 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853006 | controller | | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671' 2026-03-20 16:26:25.853010 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853014 | controller | | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B' 2026-03-20 16:26:25.853018 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853022 | controller | | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ' 2026-03-20 16:26:25.853027 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853031 | controller | | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-20 16:26:25.853035 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853039 | controller | | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E' 2026-03-20 16:26:25.853043 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853047 | controller | | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-20 16:26:25.853052 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853056 | controller | | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J' 2026-03-20 16:26:25.853060 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853064 | controller | | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU' 2026-03-20 16:26:25.853068 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853072 | controller | | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq' 2026-03-20 16:26:25.853077 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853081 | controller | | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6' 2026-03-20 16:26:25.853087 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853091 | controller | | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3' 2026-03-20 16:26:25.853095 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853099 | controller | | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk' 2026-03-20 16:26:25.853104 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853108 | controller | | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb' 2026-03-20 16:26:25.853112 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853116 | controller | | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE' 2026-03-20 16:26:25.853120 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853124 | controller | | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==' 2026-03-20 16:26:25.853149 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853154 | controller | | -----END CERTIFICATE-----''' 2026-03-20 16:26:25.853158 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853162 | controller | | ++ base64 --decode' 2026-03-20 16:26:25.853167 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853171 | controller | | ++ oc get -n watcher-kuttl-default secret cert-watcher-internal-svc -o ''jsonpath={.data.tls\.crt}''' 2026-03-20 16:26:25.853179 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853183 | controller | | + secret_cert=''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.853187 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853191 | controller | | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw' 2026-03-20 16:26:25.853196 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853200 | controller | | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx' 2026-03-20 16:26:25.853204 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853208 | controller | | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb' 2026-03-20 16:26:25.853212 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853216 | controller | | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2' 2026-03-20 16:26:25.853221 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853225 | controller | | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro' 2026-03-20 16:26:25.853229 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853233 | controller | | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671' 2026-03-20 16:26:25.853237 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853241 | controller | | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B' 2026-03-20 16:26:25.853245 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853250 | controller | | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ' 2026-03-20 16:26:25.853254 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853258 | controller | | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-20 16:26:25.853262 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853268 | controller | | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E' 2026-03-20 16:26:25.853273 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853277 | controller | | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-20 16:26:25.853281 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853285 | controller | | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J' 2026-03-20 16:26:25.853289 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853293 | controller | | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU' 2026-03-20 16:26:25.853298 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853302 | controller | | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq' 2026-03-20 16:26:25.853306 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853310 | controller | | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6' 2026-03-20 16:26:25.853314 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853318 | controller | | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3' 2026-03-20 16:26:25.853322 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853327 | controller | | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk' 2026-03-20 16:26:25.853333 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853337 | controller | | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb' 2026-03-20 16:26:25.853341 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853345 | controller | | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE' 2026-03-20 16:26:25.853349 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853353 | controller | | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==' 2026-03-20 16:26:25.853358 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853362 | controller | | -----END CERTIFICATE-----''' 2026-03-20 16:26:25.853373 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853377 | controller | | + ''['' ''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.853383 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853388 | controller | | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw' 2026-03-20 16:26:25.853392 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853396 | controller | | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx' 2026-03-20 16:26:25.853400 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853404 | controller | | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb' 2026-03-20 16:26:25.853408 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853413 | controller | | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2' 2026-03-20 16:26:25.853417 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853421 | controller | | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro' 2026-03-20 16:26:25.853425 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853430 | controller | | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671' 2026-03-20 16:26:25.853434 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853438 | controller | | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B' 2026-03-20 16:26:25.853442 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853446 | controller | | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ' 2026-03-20 16:26:25.853450 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853454 | controller | | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-20 16:26:25.853459 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853463 | controller | | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E' 2026-03-20 16:26:25.853467 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853471 | controller | | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-20 16:26:25.853475 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853479 | controller | | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J' 2026-03-20 16:26:25.853483 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853488 | controller | | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU' 2026-03-20 16:26:25.853492 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853498 | controller | | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq' 2026-03-20 16:26:25.853502 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853506 | controller | | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6' 2026-03-20 16:26:25.853510 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853515 | controller | | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3' 2026-03-20 16:26:25.853519 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853523 | controller | | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk' 2026-03-20 16:26:25.853527 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853531 | controller | | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb' 2026-03-20 16:26:25.853535 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853540 | controller | | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE' 2026-03-20 16:26:25.853544 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853548 | controller | | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==' 2026-03-20 16:26:25.853552 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853556 | controller | | -----END CERTIFICATE-----'' ''!='' ''-----BEGIN CERTIFICATE-----' 2026-03-20 16:26:25.853561 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853565 | controller | | MIIDtzCCAh+gAwIBAgIRANF4447q2MtjDeEIvUhmlVcwDQYJKoZIhvcNAQEMBQAw' 2026-03-20 16:26:25.853569 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853573 | controller | | GjEYMBYGA1UEAxMPcm9vdGNhLWludGVybmFsMB4XDTI2MDMyMDE2MTg0M1oXDTMx' 2026-03-20 16:26:25.853577 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853588 | controller | | MDMxOTE2MTg0M1owADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxb' 2026-03-20 16:26:25.853593 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853597 | controller | | NL7FiNTBRQ8Hl5cJhC584DgE2SPAXLs8WH3MsCnlC9J/iPUX9GOuAHaevAqcpzo2' 2026-03-20 16:26:25.853601 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853607 | controller | | S5hnnW69txk0LK3Sv/pS2tHNpNs8QFuam6GjK0Gmn/G9S1i4zTdHVSKsI9mhjRro' 2026-03-20 16:26:25.853611 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853616 | controller | | bn6tlPiBV5l6sCIBh/3CXZ1hgF6y12GvrWvhbUbit7QnLyegLRJdIpntFeEfN671' 2026-03-20 16:26:25.853620 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853624 | controller | | TOmyUJpV8Q6zsw2Uk0gX9yLHeN3mdmBCFJsr7GFaSi9D6pUHod+8EaJtG9O67w/B' 2026-03-20 16:26:25.853628 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853632 | controller | | 64sxP3hTRFYoL3oBRscZRz3udkfYARvpAtodS8mf5GU3JvytLt/SE1UTJryKZGMQ' 2026-03-20 16:26:25.853636 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853640 | controller | | 8MIzcAdxr0QwZb+QEkECAwEAAaOBkTCBjjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0l' 2026-03-20 16:26:25.853645 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853649 | controller | | BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS4NL6lo33E' 2026-03-20 16:26:25.853655 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853661 | controller | | 2n1gavkHzKPvAQr3TTA4BgNVHREBAf8ELjAsgip3YXRjaGVyLWludGVybmFsLndh' 2026-03-20 16:26:25.853665 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853670 | controller | | dGNoZXIta3V0dGwtZGVmYXVsdC5zdmMwDQYJKoZIhvcNAQEMBQADggGBAAPz9l1J' 2026-03-20 16:26:25.853674 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853678 | controller | | 4yvKJlSLGjV4C/+IF2fX4o8hZ/3j3j1DI49fcMp+MSJ0DVyyuXo8Y0i3ABPv3IwU' 2026-03-20 16:26:25.853682 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853686 | controller | | cTZt6NL7BFWCYtX3S0WvaoLqCGUagQuruab2bdsmHNoyvU2GDJv8IX7ScDVRIlEq' 2026-03-20 16:26:25.853690 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853707 | controller | | dKXmU3IPpQRxLgQ6lHb2ROqTLOm9gD1F0DsZqr2mJ16eNlDEYEbHtaqHls7aeCc6' 2026-03-20 16:26:25.853712 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853716 | controller | | SY+nuScf8D+5XlNv/pxbbL18b7PQrYT1MjYgvy9fYhvYguqEi3NwDbTPBJjL3cC3' 2026-03-20 16:26:25.853720 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853724 | controller | | r4alOjuTRyLPrUa+ck0nb23Q7Qj4Jb1kj8FBm4B+xSkVE5reCIIr3pDxs47J2WQk' 2026-03-20 16:26:25.853728 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853732 | controller | | TrbUjYpmx5McOX1OrnPoLVnU6ez1OOmGZljT4jcXxW7xMRoiVsYw5tVepJmqDiLb' 2026-03-20 16:26:25.853737 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853741 | controller | | kRCk/1G+eJIDLDrfK6exR27XENTcQd7HFW10+i7bhqVxfdecseih7/m/A4QwXJKE' 2026-03-20 16:26:25.853745 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853749 | controller | | ExrHi2FixsE6po2i49FFQEGANHOAQdnZyMq3gwFOuqBXHB67NyhPh7V+/g==' 2026-03-20 16:26:25.853753 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853757 | controller | | -----END CERTIFICATE-----'' '']''' 2026-03-20 16:26:25.853762 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/3-change-internal-svc-certificate 2026-03-20 16:26:25.853766 | controller | | test step completed 3-change-internal-svc-certificate' 2026-03-20 16:26:25.853770 | controller | - ' logger.go:42: 16:19:12 | watcher-tls-certs-change/4-cleanup-watcher | starting 2026-03-20 16:26:25.853774 | controller | test step 4-cleanup-watcher' 2026-03-20 16:26:25.853778 | controller | - ' logger.go:42: 16:19:19 | watcher-tls-certs-change/4-cleanup-watcher | test 2026-03-20 16:26:25.853783 | controller | step completed 4-cleanup-watcher' 2026-03-20 16:26:25.853787 | controller | - ' logger.go:42: 16:19:19 | watcher-tls-certs-change/5-clenaup-certs | starting 2026-03-20 16:26:25.853791 | controller | test step 5-clenaup-certs' 2026-03-20 16:26:25.853795 | controller | - ' logger.go:42: 16:19:19 | watcher-tls-certs-change/5-clenaup-certs | test step 2026-03-20 16:26:25.853799 | controller | completed 5-clenaup-certs' 2026-03-20 16:26:25.853804 | controller | - ' logger.go:42: 16:19:19 | watcher-tls-certs-change | skipping kubernetes event 2026-03-20 16:26:25.853808 | controller | logging' 2026-03-20 16:26:25.853812 | controller | - === CONT kuttl/harness/watcher-tls 2026-03-20 16:26:25.853816 | controller | - ' logger.go:42: 16:19:19 | watcher-tls | Skipping creation of user-supplied namespace: 2026-03-20 16:26:25.853820 | controller | watcher-kuttl-default' 2026-03-20 16:26:25.853825 | controller | - ' logger.go:42: 16:19:19 | watcher-tls/0-cleanup-watcher | starting test step 2026-03-20 16:26:25.853836 | controller | 0-cleanup-watcher' 2026-03-20 16:26:25.853840 | controller | - ' logger.go:42: 16:19:19 | watcher-tls/0-cleanup-watcher | test step completed 2026-03-20 16:26:25.853844 | controller | 0-cleanup-watcher' 2026-03-20 16:26:25.853851 | controller | - ' logger.go:42: 16:19:19 | watcher-tls/1-deploy-with-tlse | starting test step 2026-03-20 16:26:25.853855 | controller | 1-deploy-with-tlse' 2026-03-20 16:26:25.853859 | controller | - ' logger.go:42: 16:19:19 | watcher-tls/1-deploy-with-tlse | Certificate:watcher-kuttl-default/watcher-internal-svc 2026-03-20 16:26:25.853863 | controller | created' 2026-03-20 16:26:25.853868 | controller | - ' logger.go:42: 16:19:19 | watcher-tls/1-deploy-with-tlse | Certificate:watcher-kuttl-default/watcher-public-svc 2026-03-20 16:26:25.853872 | controller | created' 2026-03-20 16:26:25.853876 | controller | - ' logger.go:42: 16:19:19 | watcher-tls/1-deploy-with-tlse | Watcher:watcher-kuttl-default/watcher-kuttl 2026-03-20 16:26:25.853880 | controller | created' 2026-03-20 16:26:25.853884 | controller | - ' logger.go:42: 16:19:19 | watcher-tls/1-deploy-with-tlse | running command: 2026-03-20 16:26:25.853889 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.853893 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-20 16:26:25.853897 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-20 16:26:25.853901 | controller | - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-20 16:26:25.853905 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-20 16:26:25.853910 | controller | - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.853914 | controller | == ${SERVICEID} ]' 2026-03-20 16:26:25.853918 | controller | - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-20 16:26:25.853922 | controller | ]' 2026-03-20 16:26:25.853926 | controller | - ' [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.my\.cnf}''|base64 2026-03-20 16:26:25.853930 | controller | -d|grep -c ''ssl=1'')" == 1 ]' 2026-03-20 16:26:25.853935 | controller | - ' [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64 2026-03-20 16:26:25.853939 | controller | -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" == 2 2026-03-20 16:26:25.853943 | controller | ]' 2026-03-20 16:26:25.853947 | controller | - ' # check that both endpoints have https set' 2026-03-20 16:26:25.853951 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-20 16:26:25.853955 | controller | infra-optim | [ $(grep -c https) == 2 ]' 2026-03-20 16:26:25.853960 | controller | - ' # If we are running the container locally, skip following test' 2026-03-20 16:26:25.853964 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-20 16:26:25.853968 | controller | == "" ]; then' 2026-03-20 16:26:25.853972 | controller | - ' exit 0' 2026-03-20 16:26:25.853976 | controller | - ' fi' 2026-03-20 16:26:25.853981 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-20 16:26:25.853985 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-20 16:26:25.853989 | controller | - ' counter=0' 2026-03-20 16:26:25.853993 | controller | - ' for i in ${env_variables}; do' 2026-03-20 16:26:25.853997 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-20 16:26:25.854002 | controller | - ' echo ${i}' 2026-03-20 16:26:25.854006 | controller | - ' counter=$((counter + 1))' 2026-03-20 16:26:25.854010 | controller | - ' fi' 2026-03-20 16:26:25.854014 | controller | - ' done' 2026-03-20 16:26:25.854018 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-20 16:26:25.854023 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-20 16:26:25.854027 | controller | - ' exit 1' 2026-03-20 16:26:25.854031 | controller | - ' else' 2026-03-20 16:26:25.854035 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-20 16:26:25.854039 | controller | - ' fi' 2026-03-20 16:26:25.854043 | controller | - ' ]' 2026-03-20 16:26:25.854048 | controller | - ' logger.go:42: 16:19:19 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.854054 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-20 16:26:25.854058 | controller | - ' logger.go:42: 16:19:19 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''^watcher''' 2026-03-20 16:26:25.854062 | controller | - ' logger.go:42: 16:19:22 | watcher-tls/1-deploy-with-tlse | + ''['' 0 == 1 '']''' 2026-03-20 16:26:25.854066 | controller | - ' logger.go:42: 16:19:23 | watcher-tls/1-deploy-with-tlse | running command: 2026-03-20 16:26:25.854070 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.854075 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-20 16:26:25.854079 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-20 16:26:25.854083 | controller | - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-20 16:26:25.854087 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-20 16:26:25.854091 | controller | - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.854095 | controller | == ${SERVICEID} ]' 2026-03-20 16:26:25.854100 | controller | - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-20 16:26:25.854104 | controller | ]' 2026-03-20 16:26:25.854108 | controller | - ' [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.my\.cnf}''|base64 2026-03-20 16:26:25.854112 | controller | -d|grep -c ''ssl=1'')" == 1 ]' 2026-03-20 16:26:25.854126 | controller | - ' [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64 2026-03-20 16:26:25.854131 | controller | -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" == 2 2026-03-20 16:26:25.854135 | controller | ]' 2026-03-20 16:26:25.854139 | controller | - ' # check that both endpoints have https set' 2026-03-20 16:26:25.854143 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-20 16:26:25.854147 | controller | infra-optim | [ $(grep -c https) == 2 ]' 2026-03-20 16:26:25.854152 | controller | - ' # If we are running the container locally, skip following test' 2026-03-20 16:26:25.854156 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-20 16:26:25.854160 | controller | == "" ]; then' 2026-03-20 16:26:25.854164 | controller | - ' exit 0' 2026-03-20 16:26:25.854168 | controller | - ' fi' 2026-03-20 16:26:25.854173 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-20 16:26:25.854177 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-20 16:26:25.854181 | controller | - ' counter=0' 2026-03-20 16:26:25.854185 | controller | - ' for i in ${env_variables}; do' 2026-03-20 16:26:25.854189 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-20 16:26:25.854193 | controller | - ' echo ${i}' 2026-03-20 16:26:25.854198 | controller | - ' counter=$((counter + 1))' 2026-03-20 16:26:25.854202 | controller | - ' fi' 2026-03-20 16:26:25.854206 | controller | - ' done' 2026-03-20 16:26:25.854210 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-20 16:26:25.854214 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-20 16:26:25.854219 | controller | - ' exit 1' 2026-03-20 16:26:25.854223 | controller | - ' else' 2026-03-20 16:26:25.854227 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-20 16:26:25.854231 | controller | - ' fi' 2026-03-20 16:26:25.854235 | controller | - ' ]' 2026-03-20 16:26:25.854240 | controller | - ' logger.go:42: 16:19:23 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.854244 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-20 16:26:25.854248 | controller | - ' logger.go:42: 16:19:23 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''^watcher''' 2026-03-20 16:26:25.854254 | controller | - ' logger.go:42: 16:19:25 | watcher-tls/1-deploy-with-tlse | + ''['' 0 == 1 '']''' 2026-03-20 16:26:25.854258 | controller | - ' logger.go:42: 16:19:26 | watcher-tls/1-deploy-with-tlse | running command: 2026-03-20 16:26:25.854263 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.854267 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-20 16:26:25.854271 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-20 16:26:25.854275 | controller | - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-20 16:26:25.854279 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-20 16:26:25.854284 | controller | - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.854288 | controller | == ${SERVICEID} ]' 2026-03-20 16:26:25.854292 | controller | - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-20 16:26:25.854296 | controller | ]' 2026-03-20 16:26:25.854300 | controller | - ' [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.my\.cnf}''|base64 2026-03-20 16:26:25.854305 | controller | -d|grep -c ''ssl=1'')" == 1 ]' 2026-03-20 16:26:25.854309 | controller | - ' [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64 2026-03-20 16:26:25.854313 | controller | -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" == 2 2026-03-20 16:26:25.854317 | controller | ]' 2026-03-20 16:26:25.854321 | controller | - ' # check that both endpoints have https set' 2026-03-20 16:26:25.854326 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-20 16:26:25.854330 | controller | infra-optim | [ $(grep -c https) == 2 ]' 2026-03-20 16:26:25.854334 | controller | - ' # If we are running the container locally, skip following test' 2026-03-20 16:26:25.854338 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-20 16:26:25.854342 | controller | == "" ]; then' 2026-03-20 16:26:25.854347 | controller | - ' exit 0' 2026-03-20 16:26:25.854351 | controller | - ' fi' 2026-03-20 16:26:25.854355 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-20 16:26:25.854359 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-20 16:26:25.854363 | controller | - ' counter=0' 2026-03-20 16:26:25.854368 | controller | - ' for i in ${env_variables}; do' 2026-03-20 16:26:25.854372 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-20 16:26:25.854376 | controller | - ' echo ${i}' 2026-03-20 16:26:25.854380 | controller | - ' counter=$((counter + 1))' 2026-03-20 16:26:25.854384 | controller | - ' fi' 2026-03-20 16:26:25.854389 | controller | - ' done' 2026-03-20 16:26:25.854393 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-20 16:26:25.854397 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-20 16:26:25.854401 | controller | - ' exit 1' 2026-03-20 16:26:25.854405 | controller | - ' else' 2026-03-20 16:26:25.854409 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-20 16:26:25.854414 | controller | - ' fi' 2026-03-20 16:26:25.854418 | controller | - ' ]' 2026-03-20 16:26:25.854422 | controller | - ' logger.go:42: 16:19:26 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.854428 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-20 16:26:25.854439 | controller | - ' logger.go:42: 16:19:26 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''^watcher''' 2026-03-20 16:26:25.854443 | controller | - ' logger.go:42: 16:19:28 | watcher-tls/1-deploy-with-tlse | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.854447 | controller | - ' logger.go:42: 16:19:28 | watcher-tls/1-deploy-with-tlse | ++ grep watcher' 2026-03-20 16:26:25.854453 | controller | - ' logger.go:42: 16:19:28 | watcher-tls/1-deploy-with-tlse | ++ oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.854457 | controller | openstackclient -- openstack service list -f value -c Name -c Type -c ID' 2026-03-20 16:26:25.854462 | controller | - ' logger.go:42: 16:19:28 | watcher-tls/1-deploy-with-tlse | ++ awk ''{print $1}''' 2026-03-20 16:26:25.854466 | controller | - ' logger.go:42: 16:19:30 | watcher-tls/1-deploy-with-tlse | + SERVICEID=0bc8267151fd4fefbc0a3327396365a4' 2026-03-20 16:26:25.854470 | controller | - ' logger.go:42: 16:19:30 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.854474 | controller | keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-20 16:26:25.854478 | controller | - ' logger.go:42: 16:19:30 | watcher-tls/1-deploy-with-tlse | + ''['' 0bc8267151fd4fefbc0a3327396365a4 2026-03-20 16:26:25.854482 | controller | == 0bc8267151fd4fefbc0a3327396365a4 '']''' 2026-03-20 16:26:25.854486 | controller | - ' logger.go:42: 16:19:30 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.854491 | controller | watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-20 16:26:25.854495 | controller | - ' logger.go:42: 16:19:30 | watcher-tls/1-deploy-with-tlse | + ''['' -n '''' '']''' 2026-03-20 16:26:25.854499 | controller | - ' logger.go:42: 16:19:32 | watcher-tls/1-deploy-with-tlse | running command: 2026-03-20 16:26:25.854503 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.854508 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-20 16:26:25.854512 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-20 16:26:25.854516 | controller | - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-20 16:26:25.854520 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-20 16:26:25.854524 | controller | - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.854528 | controller | == ${SERVICEID} ]' 2026-03-20 16:26:25.854533 | controller | - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-20 16:26:25.854537 | controller | ]' 2026-03-20 16:26:25.854541 | controller | - ' [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.my\.cnf}''|base64 2026-03-20 16:26:25.854545 | controller | -d|grep -c ''ssl=1'')" == 1 ]' 2026-03-20 16:26:25.854549 | controller | - ' [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64 2026-03-20 16:26:25.854553 | controller | -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" == 2 2026-03-20 16:26:25.854558 | controller | ]' 2026-03-20 16:26:25.854562 | controller | - ' # check that both endpoints have https set' 2026-03-20 16:26:25.854566 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-20 16:26:25.854570 | controller | infra-optim | [ $(grep -c https) == 2 ]' 2026-03-20 16:26:25.854574 | controller | - ' # If we are running the container locally, skip following test' 2026-03-20 16:26:25.854579 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-20 16:26:25.854583 | controller | == "" ]; then' 2026-03-20 16:26:25.854587 | controller | - ' exit 0' 2026-03-20 16:26:25.854591 | controller | - ' fi' 2026-03-20 16:26:25.854595 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-20 16:26:25.854600 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-20 16:26:25.854604 | controller | - ' counter=0' 2026-03-20 16:26:25.854608 | controller | - ' for i in ${env_variables}; do' 2026-03-20 16:26:25.854612 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-20 16:26:25.854616 | controller | - ' echo ${i}' 2026-03-20 16:26:25.854621 | controller | - ' counter=$((counter + 1))' 2026-03-20 16:26:25.854626 | controller | - ' fi' 2026-03-20 16:26:25.854631 | controller | - ' done' 2026-03-20 16:26:25.854635 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-20 16:26:25.854639 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-20 16:26:25.854643 | controller | - ' exit 1' 2026-03-20 16:26:25.854648 | controller | - ' else' 2026-03-20 16:26:25.854652 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-20 16:26:25.854656 | controller | - ' fi' 2026-03-20 16:26:25.854660 | controller | - ' ]' 2026-03-20 16:26:25.854664 | controller | - ' logger.go:42: 16:19:32 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.854668 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-20 16:26:25.854673 | controller | - ' logger.go:42: 16:19:32 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''^watcher''' 2026-03-20 16:26:25.854677 | controller | - ' logger.go:42: 16:19:34 | watcher-tls/1-deploy-with-tlse | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.854681 | controller | - ' logger.go:42: 16:19:34 | watcher-tls/1-deploy-with-tlse | ++ grep watcher' 2026-03-20 16:26:25.854685 | controller | - ' logger.go:42: 16:19:34 | watcher-tls/1-deploy-with-tlse | ++ awk ''{print $1}''' 2026-03-20 16:26:25.854706 | controller | - ' logger.go:42: 16:19:34 | watcher-tls/1-deploy-with-tlse | ++ oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.854712 | controller | openstackclient -- openstack service list -f value -c Name -c Type -c ID' 2026-03-20 16:26:25.854717 | controller | - ' logger.go:42: 16:19:36 | watcher-tls/1-deploy-with-tlse | + SERVICEID=0bc8267151fd4fefbc0a3327396365a4' 2026-03-20 16:26:25.854721 | controller | - ' logger.go:42: 16:19:36 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.854725 | controller | keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-20 16:26:25.854729 | controller | - ' logger.go:42: 16:19:36 | watcher-tls/1-deploy-with-tlse | + ''['' 0bc8267151fd4fefbc0a3327396365a4 2026-03-20 16:26:25.854733 | controller | == 0bc8267151fd4fefbc0a3327396365a4 '']''' 2026-03-20 16:26:25.854741 | controller | - ' logger.go:42: 16:19:36 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.854745 | controller | watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-20 16:26:25.854750 | controller | - ' logger.go:42: 16:19:36 | watcher-tls/1-deploy-with-tlse | + ''['' -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.854755 | controller | '']''' 2026-03-20 16:26:25.854759 | controller | - ' logger.go:42: 16:19:36 | watcher-tls/1-deploy-with-tlse | ++ base64 -d' 2026-03-20 16:26:25.854763 | controller | - ' logger.go:42: 16:19:36 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.854767 | controller | secret watcher-kuttl-api-config-data -o ''jsonpath={.data.my\.cnf}''' 2026-03-20 16:26:25.854772 | controller | - ' logger.go:42: 16:19:36 | watcher-tls/1-deploy-with-tlse | ++ grep -c ssl=1' 2026-03-20 16:26:25.854776 | controller | - ' logger.go:42: 16:19:36 | watcher-tls/1-deploy-with-tlse | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.854782 | controller | - ' logger.go:42: 16:19:36 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''cafile 2026-03-20 16:26:25.854786 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem''' 2026-03-20 16:26:25.854790 | controller | - ' logger.go:42: 16:19:36 | watcher-tls/1-deploy-with-tlse | ++ base64 -d' 2026-03-20 16:26:25.854794 | controller | - ' logger.go:42: 16:19:36 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.854798 | controller | secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}''' 2026-03-20 16:26:25.854802 | controller | - ' logger.go:42: 16:19:37 | watcher-tls/1-deploy-with-tlse | + ''['' 2 == 2 '']''' 2026-03-20 16:26:25.854807 | controller | - ' logger.go:42: 16:19:37 | watcher-tls/1-deploy-with-tlse | + grep infra-optim' 2026-03-20 16:26:25.854811 | controller | - ' logger.go:42: 16:19:37 | watcher-tls/1-deploy-with-tlse | ++ grep -c https' 2026-03-20 16:26:25.854817 | controller | - ' logger.go:42: 16:19:37 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.854821 | controller | openstackclient -- openstack endpoint list' 2026-03-20 16:26:25.854825 | controller | - ' logger.go:42: 16:19:39 | watcher-tls/1-deploy-with-tlse | + ''['' 0 == 2 '']''' 2026-03-20 16:26:25.854830 | controller | - ' logger.go:42: 16:19:40 | watcher-tls/1-deploy-with-tlse | running command: 2026-03-20 16:26:25.854834 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.854838 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-20 16:26:25.854842 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-20 16:26:25.854848 | controller | - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-20 16:26:25.854852 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-20 16:26:25.854856 | controller | - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.854861 | controller | == ${SERVICEID} ]' 2026-03-20 16:26:25.854865 | controller | - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-20 16:26:25.854869 | controller | ]' 2026-03-20 16:26:25.854873 | controller | - ' [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.my\.cnf}''|base64 2026-03-20 16:26:25.854877 | controller | -d|grep -c ''ssl=1'')" == 1 ]' 2026-03-20 16:26:25.854882 | controller | - ' [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64 2026-03-20 16:26:25.854886 | controller | -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" == 2 2026-03-20 16:26:25.854890 | controller | ]' 2026-03-20 16:26:25.854894 | controller | - ' # check that both endpoints have https set' 2026-03-20 16:26:25.854898 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-20 16:26:25.854903 | controller | infra-optim | [ $(grep -c https) == 2 ]' 2026-03-20 16:26:25.854907 | controller | - ' # If we are running the container locally, skip following test' 2026-03-20 16:26:25.854911 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-20 16:26:25.854915 | controller | == "" ]; then' 2026-03-20 16:26:25.854919 | controller | - ' exit 0' 2026-03-20 16:26:25.854924 | controller | - ' fi' 2026-03-20 16:26:25.854928 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-20 16:26:25.854932 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-20 16:26:25.854936 | controller | - ' counter=0' 2026-03-20 16:26:25.854942 | controller | - ' for i in ${env_variables}; do' 2026-03-20 16:26:25.854953 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-20 16:26:25.854958 | controller | - ' echo ${i}' 2026-03-20 16:26:25.854962 | controller | - ' counter=$((counter + 1))' 2026-03-20 16:26:25.854967 | controller | - ' fi' 2026-03-20 16:26:25.854971 | controller | - ' done' 2026-03-20 16:26:25.854975 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-20 16:26:25.854979 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-20 16:26:25.854983 | controller | - ' exit 1' 2026-03-20 16:26:25.854987 | controller | - ' else' 2026-03-20 16:26:25.854992 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-20 16:26:25.854996 | controller | - ' fi' 2026-03-20 16:26:25.855000 | controller | - ' ]' 2026-03-20 16:26:25.855004 | controller | - ' logger.go:42: 16:19:40 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.855008 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-20 16:26:25.855013 | controller | - ' logger.go:42: 16:19:40 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''^watcher''' 2026-03-20 16:26:25.855017 | controller | - ' logger.go:42: 16:19:42 | watcher-tls/1-deploy-with-tlse | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.855023 | controller | - ' logger.go:42: 16:19:42 | watcher-tls/1-deploy-with-tlse | ++ grep watcher' 2026-03-20 16:26:25.855027 | controller | - ' logger.go:42: 16:19:42 | watcher-tls/1-deploy-with-tlse | ++ awk ''{print $1}''' 2026-03-20 16:26:25.855031 | controller | - ' logger.go:42: 16:19:42 | watcher-tls/1-deploy-with-tlse | ++ oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.855035 | controller | openstackclient -- openstack service list -f value -c Name -c Type -c ID' 2026-03-20 16:26:25.855040 | controller | - ' logger.go:42: 16:19:44 | watcher-tls/1-deploy-with-tlse | + SERVICEID=0bc8267151fd4fefbc0a3327396365a4' 2026-03-20 16:26:25.855044 | controller | - ' logger.go:42: 16:19:44 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.855048 | controller | keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-20 16:26:25.855052 | controller | - ' logger.go:42: 16:19:44 | watcher-tls/1-deploy-with-tlse | + ''['' 0bc8267151fd4fefbc0a3327396365a4 2026-03-20 16:26:25.855056 | controller | == 0bc8267151fd4fefbc0a3327396365a4 '']''' 2026-03-20 16:26:25.855060 | controller | - ' logger.go:42: 16:19:44 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.855065 | controller | watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-20 16:26:25.855069 | controller | - ' logger.go:42: 16:19:44 | watcher-tls/1-deploy-with-tlse | + ''['' -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.855073 | controller | '']''' 2026-03-20 16:26:25.855077 | controller | - ' logger.go:42: 16:19:44 | watcher-tls/1-deploy-with-tlse | ++ base64 -d' 2026-03-20 16:26:25.855082 | controller | - ' logger.go:42: 16:19:44 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.855086 | controller | secret watcher-kuttl-api-config-data -o ''jsonpath={.data.my\.cnf}''' 2026-03-20 16:26:25.855090 | controller | - ' logger.go:42: 16:19:44 | watcher-tls/1-deploy-with-tlse | ++ grep -c ssl=1' 2026-03-20 16:26:25.855094 | controller | - ' logger.go:42: 16:19:45 | watcher-tls/1-deploy-with-tlse | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.855098 | controller | - ' logger.go:42: 16:19:45 | watcher-tls/1-deploy-with-tlse | ++ base64 -d' 2026-03-20 16:26:25.855102 | controller | - ' logger.go:42: 16:19:45 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.855107 | controller | secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}''' 2026-03-20 16:26:25.855111 | controller | - ' logger.go:42: 16:19:45 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''cafile 2026-03-20 16:26:25.855115 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem''' 2026-03-20 16:26:25.855119 | controller | - ' logger.go:42: 16:19:45 | watcher-tls/1-deploy-with-tlse | + ''['' 2 == 2 '']''' 2026-03-20 16:26:25.855123 | controller | - ' logger.go:42: 16:19:45 | watcher-tls/1-deploy-with-tlse | + grep infra-optim' 2026-03-20 16:26:25.855128 | controller | - ' logger.go:42: 16:19:45 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.855132 | controller | openstackclient -- openstack endpoint list' 2026-03-20 16:26:25.855136 | controller | - ' logger.go:42: 16:19:45 | watcher-tls/1-deploy-with-tlse | ++ grep -c https' 2026-03-20 16:26:25.855140 | controller | - ' logger.go:42: 16:19:47 | watcher-tls/1-deploy-with-tlse | + ''['' 2 == 2 '']''' 2026-03-20 16:26:25.855144 | controller | - ' logger.go:42: 16:19:47 | watcher-tls/1-deploy-with-tlse | ++ oc get pods -n 2026-03-20 16:26:25.855149 | controller | openstack-operators -o name -l openstack.org/operator-name=watcher' 2026-03-20 16:26:25.855153 | controller | - ' logger.go:42: 16:19:47 | watcher-tls/1-deploy-with-tlse | + ''['' '''' == '''' 2026-03-20 16:26:25.855157 | controller | '']''' 2026-03-20 16:26:25.855162 | controller | - ' logger.go:42: 16:19:47 | watcher-tls/1-deploy-with-tlse | + exit 0' 2026-03-20 16:26:25.855166 | controller | - ' logger.go:42: 16:19:48 | watcher-tls/1-deploy-with-tlse | running command: 2026-03-20 16:26:25.855171 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.855177 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-20 16:26:25.855182 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-20 16:26:25.855186 | controller | - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-20 16:26:25.855197 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-20 16:26:25.855202 | controller | - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.855206 | controller | == ${SERVICEID} ]' 2026-03-20 16:26:25.855210 | controller | - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-20 16:26:25.855214 | controller | ]' 2026-03-20 16:26:25.855219 | controller | - ' [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.my\.cnf}''|base64 2026-03-20 16:26:25.855223 | controller | -d|grep -c ''ssl=1'')" == 1 ]' 2026-03-20 16:26:25.855227 | controller | - ' [ "$(oc get -n ${NAMESPACE} secret watcher-kuttl-api-config-data -o jsonpath=''{.data.00-default\.conf}''|base64 2026-03-20 16:26:25.855231 | controller | -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" == 2 2026-03-20 16:26:25.855235 | controller | ]' 2026-03-20 16:26:25.855240 | controller | - ' # check that both endpoints have https set' 2026-03-20 16:26:25.855244 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-20 16:26:25.855248 | controller | infra-optim | [ $(grep -c https) == 2 ]' 2026-03-20 16:26:25.855252 | controller | - ' # If we are running the container locally, skip following test' 2026-03-20 16:26:25.855256 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-20 16:26:25.855261 | controller | == "" ]; then' 2026-03-20 16:26:25.855265 | controller | - ' exit 0' 2026-03-20 16:26:25.855269 | controller | - ' fi' 2026-03-20 16:26:25.855273 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-20 16:26:25.855278 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-20 16:26:25.855282 | controller | - ' counter=0' 2026-03-20 16:26:25.855286 | controller | - ' for i in ${env_variables}; do' 2026-03-20 16:26:25.855292 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-20 16:26:25.855297 | controller | - ' echo ${i}' 2026-03-20 16:26:25.855301 | controller | - ' counter=$((counter + 1))' 2026-03-20 16:26:25.855305 | controller | - ' fi' 2026-03-20 16:26:25.855310 | controller | - ' done' 2026-03-20 16:26:25.855314 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-20 16:26:25.855318 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-20 16:26:25.855322 | controller | - ' exit 1' 2026-03-20 16:26:25.855326 | controller | - ' else' 2026-03-20 16:26:25.855331 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-20 16:26:25.855335 | controller | - ' fi' 2026-03-20 16:26:25.855339 | controller | - ' ]' 2026-03-20 16:26:25.855343 | controller | - ' logger.go:42: 16:19:48 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.855347 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-20 16:26:25.855351 | controller | - ' logger.go:42: 16:19:48 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''^watcher''' 2026-03-20 16:26:25.855355 | controller | - ' logger.go:42: 16:19:50 | watcher-tls/1-deploy-with-tlse | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.855360 | controller | - ' logger.go:42: 16:19:50 | watcher-tls/1-deploy-with-tlse | ++ oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.855364 | controller | openstackclient -- openstack service list -f value -c Name -c Type -c ID' 2026-03-20 16:26:25.855368 | controller | - ' logger.go:42: 16:19:50 | watcher-tls/1-deploy-with-tlse | ++ grep watcher' 2026-03-20 16:26:25.855372 | controller | - ' logger.go:42: 16:19:50 | watcher-tls/1-deploy-with-tlse | ++ awk ''{print $1}''' 2026-03-20 16:26:25.855378 | controller | - ' logger.go:42: 16:19:52 | watcher-tls/1-deploy-with-tlse | + SERVICEID=0bc8267151fd4fefbc0a3327396365a4' 2026-03-20 16:26:25.855383 | controller | - ' logger.go:42: 16:19:52 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.855387 | controller | keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-20 16:26:25.855391 | controller | - ' logger.go:42: 16:19:52 | watcher-tls/1-deploy-with-tlse | + ''['' 0bc8267151fd4fefbc0a3327396365a4 2026-03-20 16:26:25.855395 | controller | == 0bc8267151fd4fefbc0a3327396365a4 '']''' 2026-03-20 16:26:25.855399 | controller | - ' logger.go:42: 16:19:52 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.855404 | controller | watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-20 16:26:25.855408 | controller | - ' logger.go:42: 16:19:52 | watcher-tls/1-deploy-with-tlse | + ''['' -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.855412 | controller | '']''' 2026-03-20 16:26:25.855416 | controller | - ' logger.go:42: 16:19:52 | watcher-tls/1-deploy-with-tlse | ++ base64 -d' 2026-03-20 16:26:25.855421 | controller | - ' logger.go:42: 16:19:52 | watcher-tls/1-deploy-with-tlse | ++ grep -c ssl=1' 2026-03-20 16:26:25.855425 | controller | - ' logger.go:42: 16:19:52 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.855429 | controller | secret watcher-kuttl-api-config-data -o ''jsonpath={.data.my\.cnf}''' 2026-03-20 16:26:25.855433 | controller | - ' logger.go:42: 16:19:52 | watcher-tls/1-deploy-with-tlse | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.855437 | controller | - ' logger.go:42: 16:19:52 | watcher-tls/1-deploy-with-tlse | ++ grep -c ''cafile 2026-03-20 16:26:25.855441 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem''' 2026-03-20 16:26:25.855446 | controller | - ' logger.go:42: 16:19:52 | watcher-tls/1-deploy-with-tlse | ++ base64 -d' 2026-03-20 16:26:25.855457 | controller | - ' logger.go:42: 16:19:52 | watcher-tls/1-deploy-with-tlse | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.855461 | controller | secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}''' 2026-03-20 16:26:25.855465 | controller | - ' logger.go:42: 16:19:53 | watcher-tls/1-deploy-with-tlse | + ''['' 2 == 2 '']''' 2026-03-20 16:26:25.855470 | controller | - ' logger.go:42: 16:19:53 | watcher-tls/1-deploy-with-tlse | + grep infra-optim' 2026-03-20 16:26:25.855474 | controller | - ' logger.go:42: 16:19:53 | watcher-tls/1-deploy-with-tlse | + oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.855478 | controller | openstackclient -- openstack endpoint list' 2026-03-20 16:26:25.855482 | controller | - ' logger.go:42: 16:19:53 | watcher-tls/1-deploy-with-tlse | ++ grep -c https' 2026-03-20 16:26:25.855486 | controller | - ' logger.go:42: 16:19:55 | watcher-tls/1-deploy-with-tlse | + ''['' 2 == 2 '']''' 2026-03-20 16:26:25.855491 | controller | - ' logger.go:42: 16:19:55 | watcher-tls/1-deploy-with-tlse | ++ oc get pods -n 2026-03-20 16:26:25.855495 | controller | openstack-operators -o name -l openstack.org/operator-name=watcher' 2026-03-20 16:26:25.855499 | controller | - ' logger.go:42: 16:19:55 | watcher-tls/1-deploy-with-tlse | + ''['' '''' == '''' 2026-03-20 16:26:25.855503 | controller | '']''' 2026-03-20 16:26:25.855510 | controller | - ' logger.go:42: 16:19:55 | watcher-tls/1-deploy-with-tlse | + exit 0' 2026-03-20 16:26:25.855514 | controller | - ' logger.go:42: 16:19:55 | watcher-tls/1-deploy-with-tlse | test step completed 2026-03-20 16:26:25.855518 | controller | 1-deploy-with-tlse' 2026-03-20 16:26:25.855522 | controller | - ' logger.go:42: 16:19:55 | watcher-tls/2-patch-mtls | starting test step 2-patch-mtls' 2026-03-20 16:26:25.855527 | controller | - ' logger.go:42: 16:19:55 | watcher-tls/2-patch-mtls | running command: [sh -c 2026-03-20 16:26:25.855531 | controller | set -euxo pipefail' 2026-03-20 16:26:25.855535 | controller | - ' oc patch oscp -n $NAMESPACE openstack --type=''json'' -p=''[{"op": "replace", 2026-03-20 16:26:25.855541 | controller | "path": "/spec/memcached/templates/memcached/tls/mtls/sslVerifyMode", "value": "Request"}]''' 2026-03-20 16:26:25.855545 | controller | - ' ]' 2026-03-20 16:26:25.855551 | controller | - ' logger.go:42: 16:19:55 | watcher-tls/2-patch-mtls | + oc patch oscp -n watcher-kuttl-default 2026-03-20 16:26:25.855555 | controller | openstack --type=json ''-p=[{"op": "replace", "path": "/spec/memcached/templates/memcached/tls/mtls/sslVerifyMode", 2026-03-20 16:26:25.855559 | controller | "value": "Request"}]''' 2026-03-20 16:26:25.855564 | controller | - ' logger.go:42: 16:19:55 | watcher-tls/2-patch-mtls | openstackcontrolplane.core.openstack.org/openstack 2026-03-20 16:26:25.855568 | controller | patched' 2026-03-20 16:26:25.855572 | controller | - ' logger.go:42: 16:19:55 | watcher-tls/2-patch-mtls | running command: [sh -c 2026-03-20 16:26:25.855576 | controller | set -euxo pipefail' 2026-03-20 16:26:25.855581 | controller | - ' ' 2026-03-20 16:26:25.855585 | controller | - ' oc project ${NAMESPACE}' 2026-03-20 16:26:25.855589 | controller | - ' # Get pod names for each watcher service' 2026-03-20 16:26:25.855593 | controller | - ' APIPOD=$(oc get pods -l service=watcher-api -o jsonpath=''{.items[0].metadata.name}'')' 2026-03-20 16:26:25.855597 | controller | - ' APPLIERPOD=$(oc get pods -l service=watcher-applier -o jsonpath=''{.items[0].metadata.name}'')' 2026-03-20 16:26:25.855602 | controller | - ' DECISIONENGINEPOD=$(oc get pods -l service=watcher-decision-engine -o jsonpath=''{.items[0].metadata.name}'')' 2026-03-20 16:26:25.855606 | controller | - ' ' 2026-03-20 16:26:25.855610 | controller | - ' # Verify memcached mTLS config parameters in watcher-api config' 2026-03-20 16:26:25.855614 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.855619 | controller | - ' echo "Checking watcher-api config..."' 2026-03-20 16:26:25.855623 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.855627 | controller | | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' 2026-03-20 16:26:25.855631 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.855636 | controller | | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-20 16:26:25.855640 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.855644 | controller | | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-20 16:26:25.855648 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.855652 | controller | | grep -c "memcache_tls_enabled = true") == 1 ]' 2026-03-20 16:26:25.855657 | controller | - ' ' 2026-03-20 16:26:25.855661 | controller | - ' # Verify mTLS config parameters in memcached backend config' 2026-03-20 16:26:25.855665 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.855669 | controller | | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' 2026-03-20 16:26:25.855673 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.855678 | controller | | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-20 16:26:25.855682 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.855686 | controller | | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-20 16:26:25.855690 | controller | - ' else' 2026-03-20 16:26:25.855709 | controller | - ' exit 1' 2026-03-20 16:26:25.855713 | controller | - ' fi' 2026-03-20 16:26:25.855717 | controller | - ' ' 2026-03-20 16:26:25.855728 | controller | - ' # Verify memcached mTLS config parameters in watcher-applier config' 2026-03-20 16:26:25.855733 | controller | - ' if [ -n "${APPLIERPOD}" ]; then' 2026-03-20 16:26:25.855739 | controller | - ' echo "Checking watcher-applier config..."' 2026-03-20 16:26:25.855748 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.855752 | controller | | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' 2026-03-20 16:26:25.855756 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.855761 | controller | | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-20 16:26:25.855765 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.855787 | controller | | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-20 16:26:25.855791 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.855795 | controller | | grep -c "memcache_tls_enabled = true") == 1 ]' 2026-03-20 16:26:25.855799 | controller | - ' ' 2026-03-20 16:26:25.855804 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.855808 | controller | | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' 2026-03-20 16:26:25.855812 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.855816 | controller | | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-20 16:26:25.855820 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.855825 | controller | | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-20 16:26:25.855829 | controller | - ' else' 2026-03-20 16:26:25.855833 | controller | - ' exit 1' 2026-03-20 16:26:25.855837 | controller | - ' fi' 2026-03-20 16:26:25.855841 | controller | - ' ' 2026-03-20 16:26:25.855846 | controller | - ' # Verify memcached mTLS config parameters in watcher-decision-engine config' 2026-03-20 16:26:25.855850 | controller | - ' if [ -n "${DECISIONENGINEPOD}" ]; then' 2026-03-20 16:26:25.855854 | controller | - ' echo "Checking watcher-decision-engine config..."' 2026-03-20 16:26:25.855858 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.855863 | controller | | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' 2026-03-20 16:26:25.855867 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.855871 | controller | | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-20 16:26:25.855875 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.855879 | controller | | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-20 16:26:25.855886 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.855890 | controller | | grep -c "memcache_tls_enabled = true") == 1 ]' 2026-03-20 16:26:25.855894 | controller | - ' ' 2026-03-20 16:26:25.855898 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.855903 | controller | | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' 2026-03-20 16:26:25.855907 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.855911 | controller | | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-20 16:26:25.855915 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.855919 | controller | | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-20 16:26:25.855925 | controller | - ' else' 2026-03-20 16:26:25.855930 | controller | - ' exit 1' 2026-03-20 16:26:25.855934 | controller | - ' fi' 2026-03-20 16:26:25.855938 | controller | - ' ]' 2026-03-20 16:26:25.855942 | controller | - ' logger.go:42: 16:19:55 | watcher-tls/2-patch-mtls | + oc project watcher-kuttl-default' 2026-03-20 16:26:25.855947 | controller | - ' logger.go:42: 16:19:55 | watcher-tls/2-patch-mtls | Already on project "watcher-kuttl-default" 2026-03-20 16:26:25.855951 | controller | on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.855955 | controller | - ' logger.go:42: 16:19:55 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api 2026-03-20 16:26:25.855959 | controller | -o ''jsonpath={.items[0].metadata.name}''' 2026-03-20 16:26:25.855964 | controller | - ' logger.go:42: 16:19:55 | watcher-tls/2-patch-mtls | + APIPOD=watcher-kuttl-api-0' 2026-03-20 16:26:25.855968 | controller | - ' logger.go:42: 16:19:55 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-applier 2026-03-20 16:26:25.855972 | controller | -o ''jsonpath={.items[0].metadata.name}''' 2026-03-20 16:26:25.855976 | controller | - ' logger.go:42: 16:19:55 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0' 2026-03-20 16:26:25.855987 | controller | - ' logger.go:42: 16:19:55 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine 2026-03-20 16:26:25.855992 | controller | -o ''jsonpath={.items[0].metadata.name}''' 2026-03-20 16:26:25.855998 | controller | - ' logger.go:42: 16:19:56 | watcher-tls/2-patch-mtls | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.856003 | controller | - ' logger.go:42: 16:19:56 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-api-0 2026-03-20 16:26:25.856007 | controller | '']''' 2026-03-20 16:26:25.856011 | controller | - ' logger.go:42: 16:19:56 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-api 2026-03-20 16:26:25.856015 | controller | config...''' 2026-03-20 16:26:25.856019 | controller | - ' logger.go:42: 16:19:56 | watcher-tls/2-patch-mtls | Checking watcher-api config...' 2026-03-20 16:26:25.856024 | controller | - ' logger.go:42: 16:19:56 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile 2026-03-20 16:26:25.856028 | controller | = /etc/pki/tls/certs/mtls.crt ''' 2026-03-20 16:26:25.856032 | controller | - ' logger.go:42: 16:19:56 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api 2026-03-20 16:26:25.856036 | controller | watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.856040 | controller | - ' logger.go:42: 16:19:56 | watcher-tls/2-patch-mtls | error: Internal error occurred: 2026-03-20 16:26:25.856045 | controller | error executing command in container: container is not created or running' 2026-03-20 16:26:25.856049 | controller | - ' logger.go:42: 16:19:56 | watcher-tls/2-patch-mtls | ++ echo' 2026-03-20 16:26:25.856053 | controller | - ' logger.go:42: 16:19:56 | watcher-tls/2-patch-mtls | + ''['' 0 == 1 '']''' 2026-03-20 16:26:25.856057 | controller | - ' logger.go:42: 16:19:57 | watcher-tls/2-patch-mtls | running command: [sh -c 2026-03-20 16:26:25.856061 | controller | set -euxo pipefail' 2026-03-20 16:26:25.856066 | controller | - ' ' 2026-03-20 16:26:25.856070 | controller | - ' oc project ${NAMESPACE}' 2026-03-20 16:26:25.856074 | controller | - ' # Get pod names for each watcher service' 2026-03-20 16:26:25.856078 | controller | - ' APIPOD=$(oc get pods -l service=watcher-api -o jsonpath=''{.items[0].metadata.name}'')' 2026-03-20 16:26:25.856082 | controller | - ' APPLIERPOD=$(oc get pods -l service=watcher-applier -o jsonpath=''{.items[0].metadata.name}'')' 2026-03-20 16:26:25.856087 | controller | - ' DECISIONENGINEPOD=$(oc get pods -l service=watcher-decision-engine -o jsonpath=''{.items[0].metadata.name}'')' 2026-03-20 16:26:25.856091 | controller | - ' ' 2026-03-20 16:26:25.856095 | controller | - ' # Verify memcached mTLS config parameters in watcher-api config' 2026-03-20 16:26:25.856099 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.856103 | controller | - ' echo "Checking watcher-api config..."' 2026-03-20 16:26:25.856108 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856114 | controller | | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' 2026-03-20 16:26:25.856118 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856122 | controller | | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-20 16:26:25.856126 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856130 | controller | | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-20 16:26:25.856135 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856139 | controller | | grep -c "memcache_tls_enabled = true") == 1 ]' 2026-03-20 16:26:25.856143 | controller | - ' ' 2026-03-20 16:26:25.856147 | controller | - ' # Verify mTLS config parameters in memcached backend config' 2026-03-20 16:26:25.856151 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856156 | controller | | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' 2026-03-20 16:26:25.856160 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856164 | controller | | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-20 16:26:25.856168 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856172 | controller | | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-20 16:26:25.856177 | controller | - ' else' 2026-03-20 16:26:25.856181 | controller | - ' exit 1' 2026-03-20 16:26:25.856185 | controller | - ' fi' 2026-03-20 16:26:25.856189 | controller | - ' ' 2026-03-20 16:26:25.856194 | controller | - ' # Verify memcached mTLS config parameters in watcher-applier config' 2026-03-20 16:26:25.856198 | controller | - ' if [ -n "${APPLIERPOD}" ]; then' 2026-03-20 16:26:25.856202 | controller | - ' echo "Checking watcher-applier config..."' 2026-03-20 16:26:25.856206 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856210 | controller | | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' 2026-03-20 16:26:25.856215 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856219 | controller | | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-20 16:26:25.856223 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856234 | controller | | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-20 16:26:25.856238 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856242 | controller | | grep -c "memcache_tls_enabled = true") == 1 ]' 2026-03-20 16:26:25.856247 | controller | - ' ' 2026-03-20 16:26:25.856251 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856255 | controller | | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' 2026-03-20 16:26:25.856259 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856263 | controller | | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-20 16:26:25.856268 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856272 | controller | | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-20 16:26:25.856276 | controller | - ' else' 2026-03-20 16:26:25.856280 | controller | - ' exit 1' 2026-03-20 16:26:25.856287 | controller | - ' fi' 2026-03-20 16:26:25.856291 | controller | - ' ' 2026-03-20 16:26:25.856295 | controller | - ' # Verify memcached mTLS config parameters in watcher-decision-engine config' 2026-03-20 16:26:25.856299 | controller | - ' if [ -n "${DECISIONENGINEPOD}" ]; then' 2026-03-20 16:26:25.856303 | controller | - ' echo "Checking watcher-decision-engine config..."' 2026-03-20 16:26:25.856308 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856312 | controller | | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' 2026-03-20 16:26:25.856316 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856320 | controller | | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-20 16:26:25.856324 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856329 | controller | | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-20 16:26:25.856333 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856337 | controller | | grep -c "memcache_tls_enabled = true") == 1 ]' 2026-03-20 16:26:25.856341 | controller | - ' ' 2026-03-20 16:26:25.856345 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856350 | controller | | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' 2026-03-20 16:26:25.856354 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856358 | controller | | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-20 16:26:25.856362 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856366 | controller | | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-20 16:26:25.856370 | controller | - ' else' 2026-03-20 16:26:25.856375 | controller | - ' exit 1' 2026-03-20 16:26:25.856379 | controller | - ' fi' 2026-03-20 16:26:25.856383 | controller | - ' ]' 2026-03-20 16:26:25.856387 | controller | - ' logger.go:42: 16:19:57 | watcher-tls/2-patch-mtls | + oc project watcher-kuttl-default' 2026-03-20 16:26:25.856392 | controller | - ' logger.go:42: 16:19:57 | watcher-tls/2-patch-mtls | Already on project "watcher-kuttl-default" 2026-03-20 16:26:25.856396 | controller | on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.856400 | controller | - ' logger.go:42: 16:19:57 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api 2026-03-20 16:26:25.856404 | controller | -o ''jsonpath={.items[0].metadata.name}''' 2026-03-20 16:26:25.856409 | controller | - ' logger.go:42: 16:19:57 | watcher-tls/2-patch-mtls | + APIPOD=watcher-kuttl-api-0' 2026-03-20 16:26:25.856415 | controller | - ' logger.go:42: 16:19:57 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-applier 2026-03-20 16:26:25.856419 | controller | -o ''jsonpath={.items[0].metadata.name}''' 2026-03-20 16:26:25.856423 | controller | - ' logger.go:42: 16:19:57 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0' 2026-03-20 16:26:25.856427 | controller | - ' logger.go:42: 16:19:57 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine 2026-03-20 16:26:25.856432 | controller | -o ''jsonpath={.items[0].metadata.name}''' 2026-03-20 16:26:25.856436 | controller | - ' logger.go:42: 16:19:58 | watcher-tls/2-patch-mtls | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.856440 | controller | - ' logger.go:42: 16:19:58 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-api-0 2026-03-20 16:26:25.856444 | controller | '']''' 2026-03-20 16:26:25.856448 | controller | - ' logger.go:42: 16:19:58 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-api 2026-03-20 16:26:25.856454 | controller | config...''' 2026-03-20 16:26:25.856460 | controller | - ' logger.go:42: 16:19:58 | watcher-tls/2-patch-mtls | Checking watcher-api config...' 2026-03-20 16:26:25.856464 | controller | - ' logger.go:42: 16:19:58 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile 2026-03-20 16:26:25.856468 | controller | = /etc/pki/tls/certs/mtls.crt ''' 2026-03-20 16:26:25.856479 | controller | - ' logger.go:42: 16:19:58 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api 2026-03-20 16:26:25.856483 | controller | watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.856488 | controller | - ' logger.go:42: 16:19:58 | watcher-tls/2-patch-mtls | error: unable to upgrade 2026-03-20 16:26:25.856492 | controller | connection: container not found ("watcher-api")' 2026-03-20 16:26:25.856496 | controller | - ' logger.go:42: 16:19:58 | watcher-tls/2-patch-mtls | ++ echo' 2026-03-20 16:26:25.856500 | controller | - ' logger.go:42: 16:19:58 | watcher-tls/2-patch-mtls | + ''['' 0 == 1 '']''' 2026-03-20 16:26:25.856504 | controller | - ' logger.go:42: 16:19:59 | watcher-tls/2-patch-mtls | running command: [sh -c 2026-03-20 16:26:25.856509 | controller | set -euxo pipefail' 2026-03-20 16:26:25.856513 | controller | - ' ' 2026-03-20 16:26:25.856517 | controller | - ' oc project ${NAMESPACE}' 2026-03-20 16:26:25.856521 | controller | - ' # Get pod names for each watcher service' 2026-03-20 16:26:25.856525 | controller | - ' APIPOD=$(oc get pods -l service=watcher-api -o jsonpath=''{.items[0].metadata.name}'')' 2026-03-20 16:26:25.856529 | controller | - ' APPLIERPOD=$(oc get pods -l service=watcher-applier -o jsonpath=''{.items[0].metadata.name}'')' 2026-03-20 16:26:25.856534 | controller | - ' DECISIONENGINEPOD=$(oc get pods -l service=watcher-decision-engine -o jsonpath=''{.items[0].metadata.name}'')' 2026-03-20 16:26:25.856538 | controller | - ' ' 2026-03-20 16:26:25.856542 | controller | - ' # Verify memcached mTLS config parameters in watcher-api config' 2026-03-20 16:26:25.856546 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.856551 | controller | - ' echo "Checking watcher-api config..."' 2026-03-20 16:26:25.856555 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856559 | controller | | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' 2026-03-20 16:26:25.856563 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856567 | controller | | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-20 16:26:25.856572 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856576 | controller | | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-20 16:26:25.856580 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856584 | controller | | grep -c "memcache_tls_enabled = true") == 1 ]' 2026-03-20 16:26:25.856589 | controller | - ' ' 2026-03-20 16:26:25.856595 | controller | - ' # Verify mTLS config parameters in memcached backend config' 2026-03-20 16:26:25.856599 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856603 | controller | | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' 2026-03-20 16:26:25.856608 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856612 | controller | | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-20 16:26:25.856616 | controller | - ' [ $(echo $(oc rsh -c watcher-api $APIPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856620 | controller | | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-20 16:26:25.856624 | controller | - ' else' 2026-03-20 16:26:25.856629 | controller | - ' exit 1' 2026-03-20 16:26:25.856633 | controller | - ' fi' 2026-03-20 16:26:25.856639 | controller | - ' ' 2026-03-20 16:26:25.856643 | controller | - ' # Verify memcached mTLS config parameters in watcher-applier config' 2026-03-20 16:26:25.856647 | controller | - ' if [ -n "${APPLIERPOD}" ]; then' 2026-03-20 16:26:25.856652 | controller | - ' echo "Checking watcher-applier config..."' 2026-03-20 16:26:25.856656 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856660 | controller | | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' 2026-03-20 16:26:25.856664 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856669 | controller | | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-20 16:26:25.856673 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856677 | controller | | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-20 16:26:25.856681 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856685 | controller | | grep -c "memcache_tls_enabled = true") == 1 ]' 2026-03-20 16:26:25.856689 | controller | - ' ' 2026-03-20 16:26:25.856710 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856715 | controller | | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' 2026-03-20 16:26:25.856719 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856723 | controller | | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-20 16:26:25.856727 | controller | - ' [ $(echo $(oc rsh -c watcher-applier $APPLIERPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856738 | controller | | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-20 16:26:25.856743 | controller | - ' else' 2026-03-20 16:26:25.856747 | controller | - ' exit 1' 2026-03-20 16:26:25.856751 | controller | - ' fi' 2026-03-20 16:26:25.856755 | controller | - ' ' 2026-03-20 16:26:25.856759 | controller | - ' # Verify memcached mTLS config parameters in watcher-decision-engine config' 2026-03-20 16:26:25.856764 | controller | - ' if [ -n "${DECISIONENGINEPOD}" ]; then' 2026-03-20 16:26:25.856768 | controller | - ' echo "Checking watcher-decision-engine config..."' 2026-03-20 16:26:25.856772 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856776 | controller | | grep -c "memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt ") == 1 ]' 2026-03-20 16:26:25.856780 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856784 | controller | | grep -c "memcache_tls_keyfile = /etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-20 16:26:25.856789 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856793 | controller | | grep -c "memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-20 16:26:25.856797 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856801 | controller | | grep -c "memcache_tls_enabled = true") == 1 ]' 2026-03-20 16:26:25.856805 | controller | - ' ' 2026-03-20 16:26:25.856809 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856814 | controller | | grep -c "tls_certfile=/etc/pki/tls/certs/mtls.crt") == 1 ]' 2026-03-20 16:26:25.856818 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856826 | controller | | grep -c "tls_keyfile=/etc/pki/tls/private/mtls.key") == 1 ]' 2026-03-20 16:26:25.856831 | controller | - ' [ $(echo $(oc rsh -c watcher-decision-engine $DECISIONENGINEPOD cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.856835 | controller | | grep -c "tls_cafile=/etc/pki/tls/certs/mtls-ca.crt") == 1 ]' 2026-03-20 16:26:25.856839 | controller | - ' else' 2026-03-20 16:26:25.856843 | controller | - ' exit 1' 2026-03-20 16:26:25.856847 | controller | - ' fi' 2026-03-20 16:26:25.856852 | controller | - ' ]' 2026-03-20 16:26:25.856856 | controller | - ' logger.go:42: 16:19:59 | watcher-tls/2-patch-mtls | + oc project watcher-kuttl-default' 2026-03-20 16:26:25.856860 | controller | - ' logger.go:42: 16:19:59 | watcher-tls/2-patch-mtls | Already on project "watcher-kuttl-default" 2026-03-20 16:26:25.856864 | controller | on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.856868 | controller | - ' logger.go:42: 16:19:59 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-api 2026-03-20 16:26:25.856873 | controller | -o ''jsonpath={.items[0].metadata.name}''' 2026-03-20 16:26:25.856877 | controller | - ' logger.go:42: 16:19:59 | watcher-tls/2-patch-mtls | + APIPOD=watcher-kuttl-api-0' 2026-03-20 16:26:25.856881 | controller | - ' logger.go:42: 16:19:59 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-applier 2026-03-20 16:26:25.856885 | controller | -o ''jsonpath={.items[0].metadata.name}''' 2026-03-20 16:26:25.856889 | controller | - ' logger.go:42: 16:19:59 | watcher-tls/2-patch-mtls | + APPLIERPOD=watcher-kuttl-applier-0' 2026-03-20 16:26:25.856893 | controller | - ' logger.go:42: 16:19:59 | watcher-tls/2-patch-mtls | ++ oc get pods -l service=watcher-decision-engine 2026-03-20 16:26:25.856898 | controller | -o ''jsonpath={.items[0].metadata.name}''' 2026-03-20 16:26:25.856902 | controller | - ' logger.go:42: 16:19:59 | watcher-tls/2-patch-mtls | + DECISIONENGINEPOD=watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.856906 | controller | - ' logger.go:42: 16:19:59 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-api-0 2026-03-20 16:26:25.856910 | controller | '']''' 2026-03-20 16:26:25.856915 | controller | - ' logger.go:42: 16:19:59 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-api 2026-03-20 16:26:25.856919 | controller | config...''' 2026-03-20 16:26:25.856923 | controller | - ' logger.go:42: 16:19:59 | watcher-tls/2-patch-mtls | Checking watcher-api config...' 2026-03-20 16:26:25.856927 | controller | - ' logger.go:42: 16:19:59 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile 2026-03-20 16:26:25.856931 | controller | = /etc/pki/tls/certs/mtls.crt ''' 2026-03-20 16:26:25.856935 | controller | - ' logger.go:42: 16:19:59 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api 2026-03-20 16:26:25.856940 | controller | watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.856944 | controller | - ' logger.go:42: 16:20:00 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-20 16:26:25.856950 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.856954 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.856958 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.856969 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-20 16:26:25.856974 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-20 16:26:25.856978 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.856982 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.856986 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.856992 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.856996 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.857000 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.857005 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.857009 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.857013 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-20 16:26:25.857017 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.857022 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.857026 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-20 16:26:25.857030 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-20 16:26:25.857034 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.857038 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.857042 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.857047 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-20 16:26:25.857054 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.857058 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type 2026-03-20 16:26:25.857062 | controller | = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal 2026-03-20 16:26:25.857066 | controller | region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name 2026-03-20 16:26:25.857071 | controller | = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne 2026-03-20 16:26:25.857075 | controller | ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-20 16:26:25.857079 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-20 16:26:25.857085 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-20 16:26:25.857089 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-20 16:26:25.857093 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-20 16:26:25.857097 | controller | - ' logger.go:42: 16:20:00 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.857101 | controller | - ' logger.go:42: 16:20:00 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_keyfile 2026-03-20 16:26:25.857106 | controller | = /etc/pki/tls/private/mtls.key''' 2026-03-20 16:26:25.857110 | controller | - ' logger.go:42: 16:20:00 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api 2026-03-20 16:26:25.857114 | controller | watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.857118 | controller | - ' logger.go:42: 16:20:00 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-20 16:26:25.857122 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.857126 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.857131 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.857137 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-20 16:26:25.857141 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-20 16:26:25.857145 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.857156 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.857160 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.857164 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.857168 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.857173 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.857179 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.857183 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.857187 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-20 16:26:25.857191 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.857195 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.857200 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-20 16:26:25.857204 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-20 16:26:25.857208 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.857212 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.857216 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.857220 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-20 16:26:25.857225 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.857229 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type 2026-03-20 16:26:25.857233 | controller | = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal 2026-03-20 16:26:25.857237 | controller | region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name 2026-03-20 16:26:25.857241 | controller | = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne 2026-03-20 16:26:25.857246 | controller | ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-20 16:26:25.857250 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-20 16:26:25.857254 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-20 16:26:25.857258 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-20 16:26:25.857262 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-20 16:26:25.857266 | controller | - ' logger.go:42: 16:20:00 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.857271 | controller | - ' logger.go:42: 16:20:00 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_cafile 2026-03-20 16:26:25.857275 | controller | = /etc/pki/tls/certs/mtls-ca.crt''' 2026-03-20 16:26:25.857279 | controller | - ' logger.go:42: 16:20:00 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api 2026-03-20 16:26:25.857285 | controller | watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.857289 | controller | - ' logger.go:42: 16:20:00 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-20 16:26:25.857293 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.857298 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.857302 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.857306 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-20 16:26:25.857310 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-20 16:26:25.857314 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.857319 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.857323 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.857327 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.857331 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.857337 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.857348 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.857352 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.857356 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-20 16:26:25.857361 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.857365 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.857369 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-20 16:26:25.857373 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-20 16:26:25.857377 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.857382 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.857386 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.857392 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-20 16:26:25.857396 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.857400 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type 2026-03-20 16:26:25.857405 | controller | = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal 2026-03-20 16:26:25.857409 | controller | region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name 2026-03-20 16:26:25.857413 | controller | = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne 2026-03-20 16:26:25.857417 | controller | ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-20 16:26:25.857421 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-20 16:26:25.857426 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-20 16:26:25.857432 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-20 16:26:25.857436 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-20 16:26:25.857440 | controller | - ' logger.go:42: 16:20:00 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.857444 | controller | - ' logger.go:42: 16:20:00 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_enabled 2026-03-20 16:26:25.857448 | controller | = true''' 2026-03-20 16:26:25.857453 | controller | - ' logger.go:42: 16:20:00 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api 2026-03-20 16:26:25.857457 | controller | watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.857461 | controller | - ' logger.go:42: 16:20:00 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-20 16:26:25.857465 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.857470 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.857476 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.857480 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-20 16:26:25.857484 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-20 16:26:25.857488 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.857493 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.857497 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.857501 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.857505 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.857509 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.857514 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.857518 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.857522 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-20 16:26:25.857526 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.857536 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.857541 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-20 16:26:25.857545 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-20 16:26:25.857549 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.857553 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.857560 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.857564 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-20 16:26:25.857568 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.857573 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type 2026-03-20 16:26:25.857579 | controller | = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal 2026-03-20 16:26:25.857583 | controller | region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name 2026-03-20 16:26:25.857587 | controller | = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne 2026-03-20 16:26:25.857591 | controller | ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-20 16:26:25.857597 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-20 16:26:25.857601 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-20 16:26:25.857605 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-20 16:26:25.857609 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-20 16:26:25.857614 | controller | - ' logger.go:42: 16:20:00 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.857618 | controller | - ' logger.go:42: 16:20:00 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt' 2026-03-20 16:26:25.857622 | controller | - ' logger.go:42: 16:20:00 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api 2026-03-20 16:26:25.857626 | controller | watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.857631 | controller | - ' logger.go:42: 16:20:01 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-20 16:26:25.857635 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.857639 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.857643 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.857647 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-20 16:26:25.857652 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-20 16:26:25.857656 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.857660 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.857664 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.857668 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.857672 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.857677 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.857681 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.857685 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.857689 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-20 16:26:25.857707 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.857711 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.857715 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-20 16:26:25.857720 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-20 16:26:25.857724 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.857736 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.857740 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.857745 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-20 16:26:25.857749 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.857753 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type 2026-03-20 16:26:25.857757 | controller | = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal 2026-03-20 16:26:25.857762 | controller | region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name 2026-03-20 16:26:25.857766 | controller | = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne 2026-03-20 16:26:25.857770 | controller | ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-20 16:26:25.857774 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-20 16:26:25.857778 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-20 16:26:25.857782 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-20 16:26:25.857787 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-20 16:26:25.857791 | controller | - ' logger.go:42: 16:20:01 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.857795 | controller | - ' logger.go:42: 16:20:01 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key' 2026-03-20 16:26:25.857799 | controller | - ' logger.go:42: 16:20:01 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api 2026-03-20 16:26:25.857803 | controller | watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.857807 | controller | - ' logger.go:42: 16:20:01 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-20 16:26:25.857812 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.857816 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.857820 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.857824 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-20 16:26:25.857829 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-20 16:26:25.857833 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.857837 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.857841 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.857845 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.857849 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.857854 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.857858 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.857862 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.857868 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-20 16:26:25.857872 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.857876 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.857880 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-20 16:26:25.857887 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-20 16:26:25.857891 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.857895 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.857902 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.857906 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-20 16:26:25.857910 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.857921 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type 2026-03-20 16:26:25.857926 | controller | = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal 2026-03-20 16:26:25.857930 | controller | region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name 2026-03-20 16:26:25.857934 | controller | = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne 2026-03-20 16:26:25.857938 | controller | ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-20 16:26:25.857943 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-20 16:26:25.857947 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-20 16:26:25.857951 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-20 16:26:25.857955 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-20 16:26:25.857959 | controller | - ' logger.go:42: 16:20:01 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.857963 | controller | - ' logger.go:42: 16:20:01 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt' 2026-03-20 16:26:25.857968 | controller | - ' logger.go:42: 16:20:01 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-api 2026-03-20 16:26:25.857972 | controller | watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.857976 | controller | - ' logger.go:42: 16:20:01 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-20 16:26:25.857980 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.857984 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.857989 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.857993 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-20 16:26:25.857997 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-20 16:26:25.858001 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.858005 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.858010 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.858015 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.858020 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.858024 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.858028 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.858032 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.858036 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-20 16:26:25.858041 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.858045 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.858049 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-20 16:26:25.858053 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-20 16:26:25.858057 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.858062 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.858066 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.858070 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-20 16:26:25.858074 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.858078 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type 2026-03-20 16:26:25.858082 | controller | = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal 2026-03-20 16:26:25.858086 | controller | region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name 2026-03-20 16:26:25.858091 | controller | = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne 2026-03-20 16:26:25.858095 | controller | ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-20 16:26:25.858105 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-20 16:26:25.858110 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-20 16:26:25.858114 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-20 16:26:25.858118 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-20 16:26:25.858123 | controller | - ' logger.go:42: 16:20:01 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.858127 | controller | - ' logger.go:42: 16:20:01 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-applier-0 2026-03-20 16:26:25.858131 | controller | '']''' 2026-03-20 16:26:25.858135 | controller | - ' logger.go:42: 16:20:01 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-applier 2026-03-20 16:26:25.858139 | controller | config...''' 2026-03-20 16:26:25.858144 | controller | - ' logger.go:42: 16:20:01 | watcher-tls/2-patch-mtls | Checking watcher-applier 2026-03-20 16:26:25.858148 | controller | config...' 2026-03-20 16:26:25.858152 | controller | - ' logger.go:42: 16:20:01 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile 2026-03-20 16:26:25.858156 | controller | = /etc/pki/tls/certs/mtls.crt ''' 2026-03-20 16:26:25.858170 | controller | - ' logger.go:42: 16:20:01 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier 2026-03-20 16:26:25.858175 | controller | watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.858181 | controller | - ' logger.go:42: 16:20:02 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-20 16:26:25.858185 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.858190 | controller | control_exchange = watcher debug = True ''#'' empty notification_level means that 2026-03-20 16:26:25.858194 | controller | no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-20 16:26:25.858199 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-20 16:26:25.858203 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.858207 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.858211 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.858215 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.858220 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.858224 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.858228 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.858232 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.858236 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-20 16:26:25.858240 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.858245 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.858249 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-20 16:26:25.858253 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-20 16:26:25.858257 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.858261 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.858266 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.858270 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]'' 2026-03-20 16:26:25.858274 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-20 16:26:25.858278 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-20 16:26:25.858282 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-20 16:26:25.858287 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-20 16:26:25.858291 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-20 16:26:25.858296 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-20 16:26:25.858306 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-20 16:26:25.858310 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-20 16:26:25.858314 | controller | - ' logger.go:42: 16:20:02 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.858320 | controller | - ' logger.go:42: 16:20:02 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_keyfile 2026-03-20 16:26:25.858325 | controller | = /etc/pki/tls/private/mtls.key''' 2026-03-20 16:26:25.858329 | controller | - ' logger.go:42: 16:20:02 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier 2026-03-20 16:26:25.858333 | controller | watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.858337 | controller | - ' logger.go:42: 16:20:02 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-20 16:26:25.858341 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.858346 | controller | control_exchange = watcher debug = True ''#'' empty notification_level means that 2026-03-20 16:26:25.858350 | controller | no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-20 16:26:25.858356 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-20 16:26:25.858360 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.858365 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.858369 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.858373 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.858377 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.858381 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.858385 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.858390 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.858394 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-20 16:26:25.858398 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.858402 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.858406 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-20 16:26:25.858411 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-20 16:26:25.858415 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.858419 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.858423 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.858427 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]'' 2026-03-20 16:26:25.858431 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-20 16:26:25.858442 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-20 16:26:25.858446 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-20 16:26:25.858450 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-20 16:26:25.858454 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-20 16:26:25.858459 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-20 16:26:25.858464 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-20 16:26:25.858469 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-20 16:26:25.858473 | controller | - ' logger.go:42: 16:20:02 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.858477 | controller | - ' logger.go:42: 16:20:02 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_cafile 2026-03-20 16:26:25.858481 | controller | = /etc/pki/tls/certs/mtls-ca.crt''' 2026-03-20 16:26:25.858485 | controller | - ' logger.go:42: 16:20:02 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier 2026-03-20 16:26:25.858489 | controller | watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.858494 | controller | - ' logger.go:42: 16:20:02 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-20 16:26:25.858504 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.858509 | controller | control_exchange = watcher debug = True ''#'' empty notification_level means that 2026-03-20 16:26:25.858513 | controller | no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-20 16:26:25.858517 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-20 16:26:25.858522 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.858526 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.858530 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.858534 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.858538 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.858543 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.858547 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.858551 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.858555 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-20 16:26:25.858559 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.858563 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.858568 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-20 16:26:25.858572 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-20 16:26:25.858576 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.858580 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.858585 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.858589 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]'' 2026-03-20 16:26:25.858593 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-20 16:26:25.858597 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-20 16:26:25.858601 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-20 16:26:25.858608 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-20 16:26:25.858612 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-20 16:26:25.858616 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-20 16:26:25.858621 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-20 16:26:25.858625 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-20 16:26:25.858629 | controller | - ' logger.go:42: 16:20:02 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.858633 | controller | - ' logger.go:42: 16:20:02 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_enabled 2026-03-20 16:26:25.858637 | controller | = true''' 2026-03-20 16:26:25.858641 | controller | - ' logger.go:42: 16:20:02 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier 2026-03-20 16:26:25.858646 | controller | watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.858650 | controller | - ' logger.go:42: 16:20:02 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-20 16:26:25.858654 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.858658 | controller | control_exchange = watcher debug = True ''#'' empty notification_level means that 2026-03-20 16:26:25.858662 | controller | no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-20 16:26:25.858667 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-20 16:26:25.858676 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.858681 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.858685 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.858689 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.858708 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.858712 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.858716 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.858721 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.858725 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-20 16:26:25.858729 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.858735 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.858740 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-20 16:26:25.858744 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-20 16:26:25.858748 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.858752 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.858756 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.858762 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]'' 2026-03-20 16:26:25.858767 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-20 16:26:25.858771 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-20 16:26:25.858775 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-20 16:26:25.858779 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-20 16:26:25.858783 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-20 16:26:25.858787 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-20 16:26:25.858792 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-20 16:26:25.858796 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-20 16:26:25.858800 | controller | - ' logger.go:42: 16:20:02 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.858804 | controller | - ' logger.go:42: 16:20:02 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt' 2026-03-20 16:26:25.858808 | controller | - ' logger.go:42: 16:20:02 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier 2026-03-20 16:26:25.858813 | controller | watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.858819 | controller | - ' logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-20 16:26:25.858823 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.858827 | controller | control_exchange = watcher debug = True ''#'' empty notification_level means that 2026-03-20 16:26:25.858833 | controller | no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-20 16:26:25.858837 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-20 16:26:25.858842 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.858846 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.858850 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.858854 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.858858 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.858862 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.858873 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.858877 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.858881 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-20 16:26:25.858885 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.858890 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.858894 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-20 16:26:25.858898 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-20 16:26:25.858902 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.858908 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.858912 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.858916 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]'' 2026-03-20 16:26:25.858921 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-20 16:26:25.858925 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-20 16:26:25.858929 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-20 16:26:25.858933 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-20 16:26:25.858938 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-20 16:26:25.858942 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-20 16:26:25.858946 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-20 16:26:25.858950 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-20 16:26:25.858954 | controller | - ' logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.858959 | controller | - ' logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key' 2026-03-20 16:26:25.858963 | controller | - ' logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier 2026-03-20 16:26:25.858967 | controller | watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.858971 | controller | - ' logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-20 16:26:25.858975 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.858980 | controller | control_exchange = watcher debug = True ''#'' empty notification_level means that 2026-03-20 16:26:25.858984 | controller | no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-20 16:26:25.858988 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-20 16:26:25.858992 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.858997 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.859001 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.859005 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.859011 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.859015 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.859019 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.859024 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.859028 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-20 16:26:25.859032 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.859036 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.859043 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-20 16:26:25.859047 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-20 16:26:25.859058 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.859062 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.859067 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.859071 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]'' 2026-03-20 16:26:25.859075 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-20 16:26:25.859079 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-20 16:26:25.859083 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-20 16:26:25.859088 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-20 16:26:25.859092 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-20 16:26:25.859096 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-20 16:26:25.859100 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-20 16:26:25.859105 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-20 16:26:25.859109 | controller | - ' logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.859113 | controller | - ' logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt' 2026-03-20 16:26:25.859117 | controller | - ' logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-applier 2026-03-20 16:26:25.859121 | controller | watcher-kuttl-applier-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.859125 | controller | - ' logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-20 16:26:25.859130 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.859134 | controller | control_exchange = watcher debug = True ''#'' empty notification_level means that 2026-03-20 16:26:25.859138 | controller | no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-20 16:26:25.859142 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-20 16:26:25.859146 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.859151 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.859155 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.859159 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.859163 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.859167 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.859171 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.859176 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.859182 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-20 16:26:25.859186 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.859190 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.859194 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-20 16:26:25.859198 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-20 16:26:25.859203 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.859207 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.859211 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.859215 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[aetos_client]'' 2026-03-20 16:26:25.859220 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-20 16:26:25.859224 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-20 16:26:25.859234 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-20 16:26:25.859238 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-20 16:26:25.859242 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-20 16:26:25.859247 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-20 16:26:25.859251 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-20 16:26:25.859257 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-20 16:26:25.859261 | controller | - ' logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.859265 | controller | - ' logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | + ''['' -n watcher-kuttl-decision-engine-0 2026-03-20 16:26:25.859271 | controller | '']''' 2026-03-20 16:26:25.859275 | controller | - ' logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | + echo ''Checking watcher-decision-engine 2026-03-20 16:26:25.859281 | controller | config...''' 2026-03-20 16:26:25.859285 | controller | - ' logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | Checking watcher-decision-engine 2026-03-20 16:26:25.859289 | controller | config...' 2026-03-20 16:26:25.859294 | controller | - ' logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_certfile 2026-03-20 16:26:25.859298 | controller | = /etc/pki/tls/certs/mtls.crt ''' 2026-03-20 16:26:25.859302 | controller | - ' logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine 2026-03-20 16:26:25.859306 | controller | watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.859310 | controller | - ' logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-20 16:26:25.859314 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.859319 | controller | control_exchange = watcher debug = True ''#'' empty notification_level means that 2026-03-20 16:26:25.859323 | controller | no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-20 16:26:25.859327 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-20 16:26:25.859333 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.859337 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.859342 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.859346 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.859350 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.859354 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.859358 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.859362 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.859367 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-20 16:26:25.859371 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.859375 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.859379 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-20 16:26:25.859383 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-20 16:26:25.859388 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.859392 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.859396 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.859400 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-20 16:26:25.859404 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.859409 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type 2026-03-20 16:26:25.859413 | controller | = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal 2026-03-20 16:26:25.859417 | controller | region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name 2026-03-20 16:26:25.859421 | controller | = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne 2026-03-20 16:26:25.859431 | controller | ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-20 16:26:25.859435 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-20 16:26:25.859439 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-20 16:26:25.859444 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-20 16:26:25.859448 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-20 16:26:25.859452 | controller | - ' logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.859456 | controller | - ' logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_keyfile 2026-03-20 16:26:25.859460 | controller | = /etc/pki/tls/private/mtls.key''' 2026-03-20 16:26:25.859464 | controller | - ' logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine 2026-03-20 16:26:25.859469 | controller | watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.859473 | controller | - ' logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-20 16:26:25.859479 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.859483 | controller | control_exchange = watcher debug = True ''#'' empty notification_level means that 2026-03-20 16:26:25.859487 | controller | no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-20 16:26:25.859491 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-20 16:26:25.859496 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.859500 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.859504 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.859510 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.859514 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.859518 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.859523 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.859527 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.859531 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-20 16:26:25.859535 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.859539 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.859543 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-20 16:26:25.859549 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-20 16:26:25.859553 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.859558 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.859562 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.859566 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-20 16:26:25.859570 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.859574 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type 2026-03-20 16:26:25.859579 | controller | = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal 2026-03-20 16:26:25.859583 | controller | region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name 2026-03-20 16:26:25.859587 | controller | = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne 2026-03-20 16:26:25.859591 | controller | ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-20 16:26:25.859595 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-20 16:26:25.859599 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-20 16:26:25.859603 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-20 16:26:25.859614 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-20 16:26:25.859618 | controller | - ' logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.859624 | controller | - ' logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_cafile 2026-03-20 16:26:25.859628 | controller | = /etc/pki/tls/certs/mtls-ca.crt''' 2026-03-20 16:26:25.859633 | controller | - ' logger.go:42: 16:20:03 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine 2026-03-20 16:26:25.859637 | controller | watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.859641 | controller | - ' logger.go:42: 16:20:04 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-20 16:26:25.859645 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.859649 | controller | control_exchange = watcher debug = True ''#'' empty notification_level means that 2026-03-20 16:26:25.859654 | controller | no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-20 16:26:25.859658 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-20 16:26:25.859664 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.859668 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.859672 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.859677 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.859681 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.859685 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.859689 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.859707 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.859711 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-20 16:26:25.859715 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.859720 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.859724 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-20 16:26:25.859728 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-20 16:26:25.859732 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.859736 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.859740 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.859745 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-20 16:26:25.859749 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.859753 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type 2026-03-20 16:26:25.859757 | controller | = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal 2026-03-20 16:26:25.859761 | controller | region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name 2026-03-20 16:26:25.859765 | controller | = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne 2026-03-20 16:26:25.859771 | controller | ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-20 16:26:25.859797 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-20 16:26:25.859801 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-20 16:26:25.859805 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-20 16:26:25.859810 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-20 16:26:25.859814 | controller | - ' logger.go:42: 16:20:04 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.859818 | controller | - ' logger.go:42: 16:20:04 | watcher-tls/2-patch-mtls | ++ grep -c ''memcache_tls_enabled 2026-03-20 16:26:25.859822 | controller | = true''' 2026-03-20 16:26:25.859826 | controller | - ' logger.go:42: 16:20:04 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine 2026-03-20 16:26:25.859839 | controller | watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.859844 | controller | - ' logger.go:42: 16:20:04 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-20 16:26:25.859848 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.859853 | controller | control_exchange = watcher debug = True ''#'' empty notification_level means that 2026-03-20 16:26:25.859857 | controller | no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-20 16:26:25.859861 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-20 16:26:25.859865 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.859870 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.859874 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.859878 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.859882 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.859886 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.859890 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.859895 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.859899 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-20 16:26:25.859903 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.859907 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.859911 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-20 16:26:25.859915 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-20 16:26:25.859920 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.859924 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.859928 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.859932 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-20 16:26:25.859938 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.859942 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type 2026-03-20 16:26:25.859947 | controller | = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal 2026-03-20 16:26:25.859951 | controller | region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name 2026-03-20 16:26:25.859955 | controller | = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne 2026-03-20 16:26:25.859959 | controller | ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-20 16:26:25.859963 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-20 16:26:25.859968 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-20 16:26:25.859972 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-20 16:26:25.859976 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-20 16:26:25.859980 | controller | - ' logger.go:42: 16:20:04 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.859984 | controller | - ' logger.go:42: 16:20:04 | watcher-tls/2-patch-mtls | ++ grep -c tls_certfile=/etc/pki/tls/certs/mtls.crt' 2026-03-20 16:26:25.859988 | controller | - ' logger.go:42: 16:20:04 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine 2026-03-20 16:26:25.859992 | controller | watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.859997 | controller | - ' logger.go:42: 16:20:04 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-20 16:26:25.860001 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.860005 | controller | control_exchange = watcher debug = True ''#'' empty notification_level means that 2026-03-20 16:26:25.860018 | controller | no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-20 16:26:25.860022 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-20 16:26:25.860027 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.860031 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.860035 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.860039 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.860044 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.860048 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.860052 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.860056 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.860060 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-20 16:26:25.860064 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.860068 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.860073 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-20 16:26:25.860080 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-20 16:26:25.860084 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.860088 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.860093 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.860097 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-20 16:26:25.860101 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.860105 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type 2026-03-20 16:26:25.860109 | controller | = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal 2026-03-20 16:26:25.860113 | controller | region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name 2026-03-20 16:26:25.860118 | controller | = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne 2026-03-20 16:26:25.860122 | controller | ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-20 16:26:25.860126 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-20 16:26:25.860130 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-20 16:26:25.860134 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-20 16:26:25.860140 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-20 16:26:25.860145 | controller | - ' logger.go:42: 16:20:04 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.860149 | controller | - ' logger.go:42: 16:20:04 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine 2026-03-20 16:26:25.860153 | controller | watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.860157 | controller | - ' logger.go:42: 16:20:04 | watcher-tls/2-patch-mtls | ++ grep -c tls_keyfile=/etc/pki/tls/private/mtls.key' 2026-03-20 16:26:25.860161 | controller | - ' logger.go:42: 16:20:04 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-20 16:26:25.860167 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.860171 | controller | control_exchange = watcher debug = True ''#'' empty notification_level means that 2026-03-20 16:26:25.860175 | controller | no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-20 16:26:25.860180 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-20 16:26:25.860184 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.860194 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.860199 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.860203 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.860207 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.860212 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.860216 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.860222 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.860226 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-20 16:26:25.860230 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.860235 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.860239 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-20 16:26:25.860243 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-20 16:26:25.860247 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.860251 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.860255 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.860259 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-20 16:26:25.860264 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.860268 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type 2026-03-20 16:26:25.860272 | controller | = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal 2026-03-20 16:26:25.860276 | controller | region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name 2026-03-20 16:26:25.860280 | controller | = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne 2026-03-20 16:26:25.860284 | controller | ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-20 16:26:25.860289 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-20 16:26:25.860293 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-20 16:26:25.860297 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-20 16:26:25.860301 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-20 16:26:25.860305 | controller | - ' logger.go:42: 16:20:04 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.860310 | controller | - ' logger.go:42: 16:20:04 | watcher-tls/2-patch-mtls | ++ grep -c tls_cafile=/etc/pki/tls/certs/mtls-ca.crt' 2026-03-20 16:26:25.860314 | controller | - ' logger.go:42: 16:20:04 | watcher-tls/2-patch-mtls | +++ oc rsh -c watcher-decision-engine 2026-03-20 16:26:25.860318 | controller | watcher-kuttl-decision-engine-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.860324 | controller | - ' logger.go:42: 16:20:04 | watcher-tls/2-patch-mtls | ++ echo ''[DEFAULT]'' state_path 2026-03-20 16:26:25.860328 | controller | = /var/lib/watcher transport_url = ''rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.860334 | controller | control_exchange = watcher debug = True ''#'' empty notification_level means that 2026-03-20 16:26:25.860339 | controller | no notification will be sent notification_level = ''[database]'' connection = ''mysql+pymysql://watcher_92df:7cba4b4ad4d2f809bee67ab109ac9be9@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-20 16:26:25.860344 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-20 16:26:25.860350 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.860355 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.860375 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.860383 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.860392 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.860410 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.860417 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.860423 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.860429 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-20 16:26:25.860435 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.860441 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.860447 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-20 16:26:25.860452 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-20 16:26:25.860458 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.860464 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.860470 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.860476 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-20 16:26:25.860482 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.860486 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/ca.crt ''[aetos_client]'' endpoint_type 2026-03-20 16:26:25.860490 | controller | = internal region_name = regionOne ''[cinder_client]'' endpoint_type = internal 2026-03-20 16:26:25.860494 | controller | region_name = regionOne ''[glance_client]'' endpoint_type = internal region_name 2026-03-20 16:26:25.860498 | controller | = regionOne ''[ironic_client]'' endpoint_type = internal region_name = regionOne 2026-03-20 16:26:25.860502 | controller | ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-20 16:26:25.860506 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-20 16:26:25.860511 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-20 16:26:25.860515 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-20 16:26:25.860519 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-20 16:26:25.860523 | controller | - ' logger.go:42: 16:20:04 | watcher-tls/2-patch-mtls | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.860527 | controller | - ' logger.go:42: 16:20:04 | watcher-tls/2-patch-mtls | test step completed 2-patch-mtls' 2026-03-20 16:26:25.860532 | controller | - ' logger.go:42: 16:20:04 | watcher-tls/3-disable-podlevel-tls | starting test 2026-03-20 16:26:25.860538 | controller | step 3-disable-podlevel-tls' 2026-03-20 16:26:25.860544 | controller | - ' logger.go:42: 16:20:04 | watcher-tls/3-disable-podlevel-tls | running command: 2026-03-20 16:26:25.860550 | controller | [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type=''json'' -p=''[{"op": 2026-03-20 16:26:25.860556 | controller | "replace", "path": "/spec/apiServiceTemplate/tls/api", "value":{ "internal": {}, 2026-03-20 16:26:25.860562 | controller | "public": {} }}]''' 2026-03-20 16:26:25.860568 | controller | - ' ]' 2026-03-20 16:26:25.860574 | controller | - ' logger.go:42: 16:20:05 | watcher-tls/3-disable-podlevel-tls | watcher.watcher.openstack.org/watcher-kuttl 2026-03-20 16:26:25.860583 | controller | patched' 2026-03-20 16:26:25.860588 | controller | - ' logger.go:42: 16:20:05 | watcher-tls/3-disable-podlevel-tls | running command: 2026-03-20 16:26:25.860592 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.860597 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-20 16:26:25.860601 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-20 16:26:25.860605 | controller | - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-20 16:26:25.860609 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-20 16:26:25.860614 | controller | - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.860618 | controller | == ${SERVICEID} ]' 2026-03-20 16:26:25.860622 | controller | - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-20 16:26:25.860626 | controller | ]' 2026-03-20 16:26:25.860634 | controller | - ' # check that watcher internal endpoint does not use https' 2026-03-20 16:26:25.860638 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-20 16:26:25.860642 | controller | infra-optim | grep internal | [ $(grep -c https) == 0 ]' 2026-03-20 16:26:25.860647 | controller | - ' # check that watcher public endpoint does use https' 2026-03-20 16:26:25.860651 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-20 16:26:25.860655 | controller | infra-optim | grep public | [ $(grep -c https) == 1 ]' 2026-03-20 16:26:25.860659 | controller | - ' # If we are running the container locally, skip following test' 2026-03-20 16:26:25.860663 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-20 16:26:25.860667 | controller | == "" ]; then' 2026-03-20 16:26:25.860672 | controller | - ' exit 0' 2026-03-20 16:26:25.860686 | controller | - ' fi' 2026-03-20 16:26:25.860691 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-20 16:26:25.860711 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-20 16:26:25.860716 | controller | - ' counter=0' 2026-03-20 16:26:25.860720 | controller | - ' for i in ${env_variables}; do' 2026-03-20 16:26:25.860724 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-20 16:26:25.860728 | controller | - ' echo ${i}' 2026-03-20 16:26:25.860732 | controller | - ' counter=$((counter + 1))' 2026-03-20 16:26:25.860737 | controller | - ' fi' 2026-03-20 16:26:25.860741 | controller | - ' done' 2026-03-20 16:26:25.860745 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-20 16:26:25.860749 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-20 16:26:25.860753 | controller | - ' exit 1' 2026-03-20 16:26:25.860758 | controller | - ' else' 2026-03-20 16:26:25.860763 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-20 16:26:25.860768 | controller | - ' fi' 2026-03-20 16:26:25.860774 | controller | - ' ]' 2026-03-20 16:26:25.860780 | controller | - ' logger.go:42: 16:20:05 | watcher-tls/3-disable-podlevel-tls | + oc exec -n 2026-03-20 16:26:25.860786 | controller | watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-20 16:26:25.860792 | controller | -c Type' 2026-03-20 16:26:25.860798 | controller | - ' logger.go:42: 16:20:05 | watcher-tls/3-disable-podlevel-tls | ++ grep -c ''^watcher''' 2026-03-20 16:26:25.860803 | controller | - ' logger.go:42: 16:20:07 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 == 2026-03-20 16:26:25.860808 | controller | 1 '']''' 2026-03-20 16:26:25.860825 | controller | - ' logger.go:42: 16:20:07 | watcher-tls/3-disable-podlevel-tls | ++ grep watcher' 2026-03-20 16:26:25.860831 | controller | - ' logger.go:42: 16:20:07 | watcher-tls/3-disable-podlevel-tls | ++ oc exec -n 2026-03-20 16:26:25.860835 | controller | watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-20 16:26:25.860842 | controller | -c Type -c ID' 2026-03-20 16:26:25.860846 | controller | - ' logger.go:42: 16:20:07 | watcher-tls/3-disable-podlevel-tls | ++ awk ''{print 2026-03-20 16:26:25.860850 | controller | $1}''' 2026-03-20 16:26:25.860854 | controller | - ' logger.go:42: 16:20:10 | watcher-tls/3-disable-podlevel-tls | + SERVICEID=0bc8267151fd4fefbc0a3327396365a4' 2026-03-20 16:26:25.860860 | controller | - ' logger.go:42: 16:20:10 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n 2026-03-20 16:26:25.860866 | controller | watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-20 16:26:25.860872 | controller | - ' logger.go:42: 16:20:10 | watcher-tls/3-disable-podlevel-tls | + ''['' 0bc8267151fd4fefbc0a3327396365a4 2026-03-20 16:26:25.860878 | controller | == 0bc8267151fd4fefbc0a3327396365a4 '']''' 2026-03-20 16:26:25.860884 | controller | - ' logger.go:42: 16:20:10 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n 2026-03-20 16:26:25.860890 | controller | watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-20 16:26:25.860896 | controller | - ' logger.go:42: 16:20:10 | watcher-tls/3-disable-podlevel-tls | + ''['' -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.860902 | controller | '']''' 2026-03-20 16:26:25.860908 | controller | - ' logger.go:42: 16:20:10 | watcher-tls/3-disable-podlevel-tls | + grep internal' 2026-03-20 16:26:25.860913 | controller | - ' logger.go:42: 16:20:10 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim' 2026-03-20 16:26:25.860917 | controller | - ' logger.go:42: 16:20:10 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https' 2026-03-20 16:26:25.860921 | controller | - ' logger.go:42: 16:20:10 | watcher-tls/3-disable-podlevel-tls | + oc exec -n 2026-03-20 16:26:25.860925 | controller | watcher-kuttl-default openstackclient -- openstack endpoint list' 2026-03-20 16:26:25.860929 | controller | - ' logger.go:42: 16:20:12 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 == 2026-03-20 16:26:25.860934 | controller | 0 '']''' 2026-03-20 16:26:25.860938 | controller | - ' logger.go:42: 16:20:13 | watcher-tls/3-disable-podlevel-tls | running command: 2026-03-20 16:26:25.860942 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.860946 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-20 16:26:25.860950 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-20 16:26:25.860955 | controller | - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-20 16:26:25.860961 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-20 16:26:25.860967 | controller | - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.860973 | controller | == ${SERVICEID} ]' 2026-03-20 16:26:25.860979 | controller | - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-20 16:26:25.860985 | controller | ]' 2026-03-20 16:26:25.860991 | controller | - ' # check that watcher internal endpoint does not use https' 2026-03-20 16:26:25.860997 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-20 16:26:25.861002 | controller | infra-optim | grep internal | [ $(grep -c https) == 0 ]' 2026-03-20 16:26:25.861008 | controller | - ' # check that watcher public endpoint does use https' 2026-03-20 16:26:25.861014 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-20 16:26:25.861018 | controller | infra-optim | grep public | [ $(grep -c https) == 1 ]' 2026-03-20 16:26:25.861022 | controller | - ' # If we are running the container locally, skip following test' 2026-03-20 16:26:25.861036 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-20 16:26:25.861041 | controller | == "" ]; then' 2026-03-20 16:26:25.861046 | controller | - ' exit 0' 2026-03-20 16:26:25.861050 | controller | - ' fi' 2026-03-20 16:26:25.861054 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-20 16:26:25.861061 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-20 16:26:25.861069 | controller | - ' counter=0' 2026-03-20 16:26:25.861073 | controller | - ' for i in ${env_variables}; do' 2026-03-20 16:26:25.861077 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-20 16:26:25.861081 | controller | - ' echo ${i}' 2026-03-20 16:26:25.861086 | controller | - ' counter=$((counter + 1))' 2026-03-20 16:26:25.861090 | controller | - ' fi' 2026-03-20 16:26:25.861094 | controller | - ' done' 2026-03-20 16:26:25.861098 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-20 16:26:25.861102 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-20 16:26:25.861107 | controller | - ' exit 1' 2026-03-20 16:26:25.861111 | controller | - ' else' 2026-03-20 16:26:25.861115 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-20 16:26:25.861119 | controller | - ' fi' 2026-03-20 16:26:25.861123 | controller | - ' ]' 2026-03-20 16:26:25.861128 | controller | - ' logger.go:42: 16:20:13 | watcher-tls/3-disable-podlevel-tls | + oc exec -n 2026-03-20 16:26:25.861132 | controller | watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-20 16:26:25.861136 | controller | -c Type' 2026-03-20 16:26:25.861140 | controller | - ' logger.go:42: 16:20:13 | watcher-tls/3-disable-podlevel-tls | ++ grep -c ''^watcher''' 2026-03-20 16:26:25.861144 | controller | - ' logger.go:42: 16:20:15 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 == 2026-03-20 16:26:25.861148 | controller | 1 '']''' 2026-03-20 16:26:25.861153 | controller | - ' logger.go:42: 16:20:15 | watcher-tls/3-disable-podlevel-tls | ++ oc exec -n 2026-03-20 16:26:25.861157 | controller | watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-20 16:26:25.861161 | controller | -c Type -c ID' 2026-03-20 16:26:25.861166 | controller | - ' logger.go:42: 16:20:15 | watcher-tls/3-disable-podlevel-tls | ++ awk ''{print 2026-03-20 16:26:25.861170 | controller | $1}''' 2026-03-20 16:26:25.861174 | controller | - ' logger.go:42: 16:20:15 | watcher-tls/3-disable-podlevel-tls | ++ grep watcher' 2026-03-20 16:26:25.861178 | controller | - ' logger.go:42: 16:20:17 | watcher-tls/3-disable-podlevel-tls | + SERVICEID=0bc8267151fd4fefbc0a3327396365a4' 2026-03-20 16:26:25.861182 | controller | - ' logger.go:42: 16:20:17 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n 2026-03-20 16:26:25.861186 | controller | watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-20 16:26:25.861191 | controller | - ' logger.go:42: 16:20:18 | watcher-tls/3-disable-podlevel-tls | + ''['' 0bc8267151fd4fefbc0a3327396365a4 2026-03-20 16:26:25.861195 | controller | == 0bc8267151fd4fefbc0a3327396365a4 '']''' 2026-03-20 16:26:25.861199 | controller | - ' logger.go:42: 16:20:18 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n 2026-03-20 16:26:25.861203 | controller | watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-20 16:26:25.861207 | controller | - ' logger.go:42: 16:20:18 | watcher-tls/3-disable-podlevel-tls | + ''['' -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.861212 | controller | '']''' 2026-03-20 16:26:25.861216 | controller | - ' logger.go:42: 16:20:18 | watcher-tls/3-disable-podlevel-tls | + grep internal' 2026-03-20 16:26:25.861222 | controller | - ' logger.go:42: 16:20:18 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim' 2026-03-20 16:26:25.861226 | controller | - ' logger.go:42: 16:20:18 | watcher-tls/3-disable-podlevel-tls | + oc exec -n 2026-03-20 16:26:25.861230 | controller | watcher-kuttl-default openstackclient -- openstack endpoint list' 2026-03-20 16:26:25.861234 | controller | - ' logger.go:42: 16:20:18 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https' 2026-03-20 16:26:25.861238 | controller | - ' logger.go:42: 16:20:20 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 == 2026-03-20 16:26:25.861243 | controller | 0 '']''' 2026-03-20 16:26:25.861249 | controller | - ' logger.go:42: 16:20:21 | watcher-tls/3-disable-podlevel-tls | running command: 2026-03-20 16:26:25.861253 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.861257 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-20 16:26:25.861261 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-20 16:26:25.861266 | controller | - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-20 16:26:25.861271 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-20 16:26:25.861276 | controller | - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.861280 | controller | == ${SERVICEID} ]' 2026-03-20 16:26:25.861287 | controller | - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-20 16:26:25.861291 | controller | ]' 2026-03-20 16:26:25.861295 | controller | - ' # check that watcher internal endpoint does not use https' 2026-03-20 16:26:25.861300 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-20 16:26:25.861304 | controller | infra-optim | grep internal | [ $(grep -c https) == 0 ]' 2026-03-20 16:26:25.861308 | controller | - ' # check that watcher public endpoint does use https' 2026-03-20 16:26:25.861312 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-20 16:26:25.861324 | controller | infra-optim | grep public | [ $(grep -c https) == 1 ]' 2026-03-20 16:26:25.861329 | controller | - ' # If we are running the container locally, skip following test' 2026-03-20 16:26:25.861333 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-20 16:26:25.861337 | controller | == "" ]; then' 2026-03-20 16:26:25.861341 | controller | - ' exit 0' 2026-03-20 16:26:25.861346 | controller | - ' fi' 2026-03-20 16:26:25.861350 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-20 16:26:25.861354 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-20 16:26:25.861358 | controller | - ' counter=0' 2026-03-20 16:26:25.861363 | controller | - ' for i in ${env_variables}; do' 2026-03-20 16:26:25.861367 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-20 16:26:25.861371 | controller | - ' echo ${i}' 2026-03-20 16:26:25.861375 | controller | - ' counter=$((counter + 1))' 2026-03-20 16:26:25.861379 | controller | - ' fi' 2026-03-20 16:26:25.861383 | controller | - ' done' 2026-03-20 16:26:25.861387 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-20 16:26:25.861392 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-20 16:26:25.861396 | controller | - ' exit 1' 2026-03-20 16:26:25.861400 | controller | - ' else' 2026-03-20 16:26:25.861404 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-20 16:26:25.861408 | controller | - ' fi' 2026-03-20 16:26:25.861413 | controller | - ' ]' 2026-03-20 16:26:25.861417 | controller | - ' logger.go:42: 16:20:21 | watcher-tls/3-disable-podlevel-tls | + oc exec -n 2026-03-20 16:26:25.861421 | controller | watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-20 16:26:25.861425 | controller | -c Type' 2026-03-20 16:26:25.861429 | controller | - ' logger.go:42: 16:20:21 | watcher-tls/3-disable-podlevel-tls | ++ grep -c ''^watcher''' 2026-03-20 16:26:25.861434 | controller | - ' logger.go:42: 16:20:23 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 == 2026-03-20 16:26:25.861440 | controller | 1 '']''' 2026-03-20 16:26:25.861446 | controller | - ' logger.go:42: 16:20:23 | watcher-tls/3-disable-podlevel-tls | ++ grep watcher' 2026-03-20 16:26:25.861451 | controller | - ' logger.go:42: 16:20:23 | watcher-tls/3-disable-podlevel-tls | ++ oc exec -n 2026-03-20 16:26:25.861457 | controller | watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-20 16:26:25.861463 | controller | -c Type -c ID' 2026-03-20 16:26:25.861467 | controller | - ' logger.go:42: 16:20:23 | watcher-tls/3-disable-podlevel-tls | ++ awk ''{print 2026-03-20 16:26:25.861472 | controller | $1}''' 2026-03-20 16:26:25.861476 | controller | - ' logger.go:42: 16:20:25 | watcher-tls/3-disable-podlevel-tls | + SERVICEID=0bc8267151fd4fefbc0a3327396365a4' 2026-03-20 16:26:25.861480 | controller | - ' logger.go:42: 16:20:25 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n 2026-03-20 16:26:25.861484 | controller | watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-20 16:26:25.861489 | controller | - ' logger.go:42: 16:20:25 | watcher-tls/3-disable-podlevel-tls | + ''['' 0bc8267151fd4fefbc0a3327396365a4 2026-03-20 16:26:25.861493 | controller | == 0bc8267151fd4fefbc0a3327396365a4 '']''' 2026-03-20 16:26:25.861497 | controller | - ' logger.go:42: 16:20:25 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n 2026-03-20 16:26:25.861501 | controller | watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-20 16:26:25.861506 | controller | - ' logger.go:42: 16:20:25 | watcher-tls/3-disable-podlevel-tls | + ''['' -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.861510 | controller | '']''' 2026-03-20 16:26:25.861514 | controller | - ' logger.go:42: 16:20:25 | watcher-tls/3-disable-podlevel-tls | + grep internal' 2026-03-20 16:26:25.861518 | controller | - ' logger.go:42: 16:20:25 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https' 2026-03-20 16:26:25.861523 | controller | - ' logger.go:42: 16:20:25 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim' 2026-03-20 16:26:25.861527 | controller | - ' logger.go:42: 16:20:25 | watcher-tls/3-disable-podlevel-tls | + oc exec -n 2026-03-20 16:26:25.861531 | controller | watcher-kuttl-default openstackclient -- openstack endpoint list' 2026-03-20 16:26:25.861535 | controller | - ' logger.go:42: 16:20:27 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 == 2026-03-20 16:26:25.861539 | controller | 0 '']''' 2026-03-20 16:26:25.861543 | controller | - ' logger.go:42: 16:20:28 | watcher-tls/3-disable-podlevel-tls | running command: 2026-03-20 16:26:25.861548 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.861552 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-20 16:26:25.861556 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-20 16:26:25.861560 | controller | - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-20 16:26:25.861564 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-20 16:26:25.861568 | controller | - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.861573 | controller | == ${SERVICEID} ]' 2026-03-20 16:26:25.861577 | controller | - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-20 16:26:25.861581 | controller | ]' 2026-03-20 16:26:25.861585 | controller | - ' # check that watcher internal endpoint does not use https' 2026-03-20 16:26:25.861589 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-20 16:26:25.861594 | controller | infra-optim | grep internal | [ $(grep -c https) == 0 ]' 2026-03-20 16:26:25.861605 | controller | - ' # check that watcher public endpoint does use https' 2026-03-20 16:26:25.861610 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-20 16:26:25.861615 | controller | infra-optim | grep public | [ $(grep -c https) == 1 ]' 2026-03-20 16:26:25.861621 | controller | - ' # If we are running the container locally, skip following test' 2026-03-20 16:26:25.861626 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-20 16:26:25.861631 | controller | == "" ]; then' 2026-03-20 16:26:25.861635 | controller | - ' exit 0' 2026-03-20 16:26:25.861639 | controller | - ' fi' 2026-03-20 16:26:25.861645 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-20 16:26:25.861660 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-20 16:26:25.861664 | controller | - ' counter=0' 2026-03-20 16:26:25.861669 | controller | - ' for i in ${env_variables}; do' 2026-03-20 16:26:25.861673 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-20 16:26:25.861677 | controller | - ' echo ${i}' 2026-03-20 16:26:25.861681 | controller | - ' counter=$((counter + 1))' 2026-03-20 16:26:25.861685 | controller | - ' fi' 2026-03-20 16:26:25.861690 | controller | - ' done' 2026-03-20 16:26:25.861758 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-20 16:26:25.861765 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-20 16:26:25.861771 | controller | - ' exit 1' 2026-03-20 16:26:25.861776 | controller | - ' else' 2026-03-20 16:26:25.861782 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-20 16:26:25.861787 | controller | - ' fi' 2026-03-20 16:26:25.861792 | controller | - ' ]' 2026-03-20 16:26:25.861798 | controller | - ' logger.go:42: 16:20:28 | watcher-tls/3-disable-podlevel-tls | + oc exec -n 2026-03-20 16:26:25.861803 | controller | watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-20 16:26:25.861808 | controller | -c Type' 2026-03-20 16:26:25.861814 | controller | - ' logger.go:42: 16:20:28 | watcher-tls/3-disable-podlevel-tls | ++ grep -c ''^watcher''' 2026-03-20 16:26:25.861819 | controller | - ' logger.go:42: 16:20:30 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 == 2026-03-20 16:26:25.861824 | controller | 1 '']''' 2026-03-20 16:26:25.861828 | controller | - ' logger.go:42: 16:20:30 | watcher-tls/3-disable-podlevel-tls | ++ oc exec -n 2026-03-20 16:26:25.861832 | controller | watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-20 16:26:25.861836 | controller | -c Type -c ID' 2026-03-20 16:26:25.861840 | controller | - ' logger.go:42: 16:20:30 | watcher-tls/3-disable-podlevel-tls | ++ awk ''{print 2026-03-20 16:26:25.861845 | controller | $1}''' 2026-03-20 16:26:25.861849 | controller | - ' logger.go:42: 16:20:30 | watcher-tls/3-disable-podlevel-tls | ++ grep watcher' 2026-03-20 16:26:25.861853 | controller | - ' logger.go:42: 16:20:32 | watcher-tls/3-disable-podlevel-tls | + SERVICEID=0bc8267151fd4fefbc0a3327396365a4' 2026-03-20 16:26:25.861857 | controller | - ' logger.go:42: 16:20:32 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n 2026-03-20 16:26:25.861862 | controller | watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-20 16:26:25.861866 | controller | - ' logger.go:42: 16:20:32 | watcher-tls/3-disable-podlevel-tls | + ''['' 0bc8267151fd4fefbc0a3327396365a4 2026-03-20 16:26:25.861870 | controller | == 0bc8267151fd4fefbc0a3327396365a4 '']''' 2026-03-20 16:26:25.861874 | controller | - ' logger.go:42: 16:20:32 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n 2026-03-20 16:26:25.861878 | controller | watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-20 16:26:25.861883 | controller | - ' logger.go:42: 16:20:33 | watcher-tls/3-disable-podlevel-tls | + ''['' -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.861887 | controller | '']''' 2026-03-20 16:26:25.861891 | controller | - ' logger.go:42: 16:20:33 | watcher-tls/3-disable-podlevel-tls | + grep internal' 2026-03-20 16:26:25.861895 | controller | - ' logger.go:42: 16:20:33 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim' 2026-03-20 16:26:25.861899 | controller | - ' logger.go:42: 16:20:33 | watcher-tls/3-disable-podlevel-tls | + oc exec -n 2026-03-20 16:26:25.861904 | controller | watcher-kuttl-default openstackclient -- openstack endpoint list' 2026-03-20 16:26:25.861908 | controller | - ' logger.go:42: 16:20:33 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https' 2026-03-20 16:26:25.861912 | controller | - ' logger.go:42: 16:20:34 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 == 2026-03-20 16:26:25.861916 | controller | 0 '']''' 2026-03-20 16:26:25.861920 | controller | - ' logger.go:42: 16:20:36 | watcher-tls/3-disable-podlevel-tls | running command: 2026-03-20 16:26:25.861927 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.861932 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-20 16:26:25.861936 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-20 16:26:25.861940 | controller | - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-20 16:26:25.861944 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-20 16:26:25.861948 | controller | - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.861952 | controller | == ${SERVICEID} ]' 2026-03-20 16:26:25.861957 | controller | - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-20 16:26:25.861961 | controller | ]' 2026-03-20 16:26:25.861965 | controller | - ' # check that watcher internal endpoint does not use https' 2026-03-20 16:26:25.861979 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-20 16:26:25.861984 | controller | infra-optim | grep internal | [ $(grep -c https) == 0 ]' 2026-03-20 16:26:25.861988 | controller | - ' # check that watcher public endpoint does use https' 2026-03-20 16:26:25.861992 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-20 16:26:25.861996 | controller | infra-optim | grep public | [ $(grep -c https) == 1 ]' 2026-03-20 16:26:25.862000 | controller | - ' # If we are running the container locally, skip following test' 2026-03-20 16:26:25.862004 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-20 16:26:25.862009 | controller | == "" ]; then' 2026-03-20 16:26:25.862013 | controller | - ' exit 0' 2026-03-20 16:26:25.862017 | controller | - ' fi' 2026-03-20 16:26:25.862021 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-20 16:26:25.862025 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-20 16:26:25.862029 | controller | - ' counter=0' 2026-03-20 16:26:25.862034 | controller | - ' for i in ${env_variables}; do' 2026-03-20 16:26:25.862038 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-20 16:26:25.862042 | controller | - ' echo ${i}' 2026-03-20 16:26:25.862046 | controller | - ' counter=$((counter + 1))' 2026-03-20 16:26:25.862050 | controller | - ' fi' 2026-03-20 16:26:25.862054 | controller | - ' done' 2026-03-20 16:26:25.862059 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-20 16:26:25.862063 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-20 16:26:25.862067 | controller | - ' exit 1' 2026-03-20 16:26:25.862071 | controller | - ' else' 2026-03-20 16:26:25.862075 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-20 16:26:25.862082 | controller | - ' fi' 2026-03-20 16:26:25.862086 | controller | - ' ]' 2026-03-20 16:26:25.862090 | controller | - ' logger.go:42: 16:20:36 | watcher-tls/3-disable-podlevel-tls | + oc exec -n 2026-03-20 16:26:25.862095 | controller | watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-20 16:26:25.862099 | controller | -c Type' 2026-03-20 16:26:25.862103 | controller | - ' logger.go:42: 16:20:36 | watcher-tls/3-disable-podlevel-tls | ++ grep -c ''^watcher''' 2026-03-20 16:26:25.862107 | controller | - ' logger.go:42: 16:20:38 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 == 2026-03-20 16:26:25.862111 | controller | 1 '']''' 2026-03-20 16:26:25.862116 | controller | - ' logger.go:42: 16:20:38 | watcher-tls/3-disable-podlevel-tls | ++ grep watcher' 2026-03-20 16:26:25.862120 | controller | - ' logger.go:42: 16:20:38 | watcher-tls/3-disable-podlevel-tls | ++ oc exec -n 2026-03-20 16:26:25.862124 | controller | watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-20 16:26:25.862128 | controller | -c Type -c ID' 2026-03-20 16:26:25.862142 | controller | - ' logger.go:42: 16:20:38 | watcher-tls/3-disable-podlevel-tls | ++ awk ''{print 2026-03-20 16:26:25.862165 | controller | $1}''' 2026-03-20 16:26:25.862171 | controller | - ' logger.go:42: 16:20:40 | watcher-tls/3-disable-podlevel-tls | + SERVICEID=0bc8267151fd4fefbc0a3327396365a4' 2026-03-20 16:26:25.862175 | controller | - ' logger.go:42: 16:20:40 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n 2026-03-20 16:26:25.862181 | controller | watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-20 16:26:25.862185 | controller | - ' logger.go:42: 16:20:40 | watcher-tls/3-disable-podlevel-tls | + ''['' 0bc8267151fd4fefbc0a3327396365a4 2026-03-20 16:26:25.862190 | controller | == 0bc8267151fd4fefbc0a3327396365a4 '']''' 2026-03-20 16:26:25.862194 | controller | - ' logger.go:42: 16:20:40 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n 2026-03-20 16:26:25.862198 | controller | watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-20 16:26:25.862202 | controller | - ' logger.go:42: 16:20:40 | watcher-tls/3-disable-podlevel-tls | + ''['' -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.862206 | controller | '']''' 2026-03-20 16:26:25.862211 | controller | - ' logger.go:42: 16:20:40 | watcher-tls/3-disable-podlevel-tls | + grep internal' 2026-03-20 16:26:25.862215 | controller | - ' logger.go:42: 16:20:40 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim' 2026-03-20 16:26:25.862219 | controller | - ' logger.go:42: 16:20:40 | watcher-tls/3-disable-podlevel-tls | + oc exec -n 2026-03-20 16:26:25.862223 | controller | watcher-kuttl-default openstackclient -- openstack endpoint list' 2026-03-20 16:26:25.862227 | controller | - ' logger.go:42: 16:20:40 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https' 2026-03-20 16:26:25.862231 | controller | - ' logger.go:42: 16:20:47 | watcher-tls/3-disable-podlevel-tls | + ''['' 0 == 2026-03-20 16:26:25.862236 | controller | 0 '']''' 2026-03-20 16:26:25.862240 | controller | - ' logger.go:42: 16:20:47 | watcher-tls/3-disable-podlevel-tls | + grep public' 2026-03-20 16:26:25.862244 | controller | - ' logger.go:42: 16:20:47 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim' 2026-03-20 16:26:25.862248 | controller | - ' logger.go:42: 16:20:47 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https' 2026-03-20 16:26:25.862252 | controller | - ' logger.go:42: 16:20:47 | watcher-tls/3-disable-podlevel-tls | + oc exec -n 2026-03-20 16:26:25.862257 | controller | watcher-kuttl-default openstackclient -- openstack endpoint list' 2026-03-20 16:26:25.862261 | controller | - ' logger.go:42: 16:20:49 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 == 2026-03-20 16:26:25.862265 | controller | 1 '']''' 2026-03-20 16:26:25.862269 | controller | - ' logger.go:42: 16:20:49 | watcher-tls/3-disable-podlevel-tls | ++ oc get pods 2026-03-20 16:26:25.862280 | controller | -n openstack-operators -o name -l openstack.org/operator-name=watcher' 2026-03-20 16:26:25.862285 | controller | - ' logger.go:42: 16:20:49 | watcher-tls/3-disable-podlevel-tls | + ''['' '''' 2026-03-20 16:26:25.862289 | controller | == '''' '']''' 2026-03-20 16:26:25.862293 | controller | - ' logger.go:42: 16:20:49 | watcher-tls/3-disable-podlevel-tls | + exit 0' 2026-03-20 16:26:25.862297 | controller | - ' logger.go:42: 16:20:51 | watcher-tls/3-disable-podlevel-tls | running command: 2026-03-20 16:26:25.862301 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.862306 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-20 16:26:25.862310 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-20 16:26:25.862314 | controller | - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-20 16:26:25.862318 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-20 16:26:25.862322 | controller | - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.862327 | controller | == ${SERVICEID} ]' 2026-03-20 16:26:25.862331 | controller | - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-20 16:26:25.862343 | controller | ]' 2026-03-20 16:26:25.862347 | controller | - ' # check that watcher internal endpoint does not use https' 2026-03-20 16:26:25.862352 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-20 16:26:25.862357 | controller | infra-optim | grep internal | [ $(grep -c https) == 0 ]' 2026-03-20 16:26:25.862363 | controller | - ' # check that watcher public endpoint does use https' 2026-03-20 16:26:25.862369 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-20 16:26:25.862374 | controller | infra-optim | grep public | [ $(grep -c https) == 1 ]' 2026-03-20 16:26:25.862382 | controller | - ' # If we are running the container locally, skip following test' 2026-03-20 16:26:25.862388 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-20 16:26:25.862393 | controller | == "" ]; then' 2026-03-20 16:26:25.862399 | controller | - ' exit 0' 2026-03-20 16:26:25.862404 | controller | - ' fi' 2026-03-20 16:26:25.862410 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-20 16:26:25.862415 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-20 16:26:25.862421 | controller | - ' counter=0' 2026-03-20 16:26:25.862426 | controller | - ' for i in ${env_variables}; do' 2026-03-20 16:26:25.862431 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-20 16:26:25.862437 | controller | - ' echo ${i}' 2026-03-20 16:26:25.862442 | controller | - ' counter=$((counter + 1))' 2026-03-20 16:26:25.862447 | controller | - ' fi' 2026-03-20 16:26:25.862453 | controller | - ' done' 2026-03-20 16:26:25.862458 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-20 16:26:25.862463 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-20 16:26:25.862469 | controller | - ' exit 1' 2026-03-20 16:26:25.862474 | controller | - ' else' 2026-03-20 16:26:25.862480 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-20 16:26:25.862486 | controller | - ' fi' 2026-03-20 16:26:25.862492 | controller | - ' ]' 2026-03-20 16:26:25.862497 | controller | - ' logger.go:42: 16:20:51 | watcher-tls/3-disable-podlevel-tls | + oc exec -n 2026-03-20 16:26:25.862503 | controller | watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-20 16:26:25.862508 | controller | -c Type' 2026-03-20 16:26:25.862514 | controller | - ' logger.go:42: 16:20:51 | watcher-tls/3-disable-podlevel-tls | ++ grep -c ''^watcher''' 2026-03-20 16:26:25.862520 | controller | - ' logger.go:42: 16:20:53 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 == 2026-03-20 16:26:25.862525 | controller | 1 '']''' 2026-03-20 16:26:25.862531 | controller | - ' logger.go:42: 16:20:53 | watcher-tls/3-disable-podlevel-tls | ++ grep watcher' 2026-03-20 16:26:25.862536 | controller | - ' logger.go:42: 16:20:53 | watcher-tls/3-disable-podlevel-tls | ++ awk ''{print 2026-03-20 16:26:25.862542 | controller | $1}''' 2026-03-20 16:26:25.862547 | controller | - ' logger.go:42: 16:20:53 | watcher-tls/3-disable-podlevel-tls | ++ oc exec -n 2026-03-20 16:26:25.862552 | controller | watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-20 16:26:25.862558 | controller | -c Type -c ID' 2026-03-20 16:26:25.862563 | controller | - ' logger.go:42: 16:20:55 | watcher-tls/3-disable-podlevel-tls | + SERVICEID=0bc8267151fd4fefbc0a3327396365a4' 2026-03-20 16:26:25.862569 | controller | - ' logger.go:42: 16:20:55 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n 2026-03-20 16:26:25.862574 | controller | watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-20 16:26:25.862580 | controller | - ' logger.go:42: 16:20:55 | watcher-tls/3-disable-podlevel-tls | + ''['' 0bc8267151fd4fefbc0a3327396365a4 2026-03-20 16:26:25.862585 | controller | == 0bc8267151fd4fefbc0a3327396365a4 '']''' 2026-03-20 16:26:25.862592 | controller | - ' logger.go:42: 16:20:55 | watcher-tls/3-disable-podlevel-tls | ++ oc get -n 2026-03-20 16:26:25.862599 | controller | watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-20 16:26:25.862604 | controller | - ' logger.go:42: 16:20:55 | watcher-tls/3-disable-podlevel-tls | + ''['' -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.862608 | controller | '']''' 2026-03-20 16:26:25.862613 | controller | - ' logger.go:42: 16:20:55 | watcher-tls/3-disable-podlevel-tls | + grep internal' 2026-03-20 16:26:25.862627 | controller | - ' logger.go:42: 16:20:55 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim' 2026-03-20 16:26:25.862633 | controller | - ' logger.go:42: 16:20:55 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https' 2026-03-20 16:26:25.862638 | controller | - ' logger.go:42: 16:20:55 | watcher-tls/3-disable-podlevel-tls | + oc exec -n 2026-03-20 16:26:25.862644 | controller | watcher-kuttl-default openstackclient -- openstack endpoint list' 2026-03-20 16:26:25.862650 | controller | - ' logger.go:42: 16:20:57 | watcher-tls/3-disable-podlevel-tls | + ''['' 0 == 2026-03-20 16:26:25.862654 | controller | 0 '']''' 2026-03-20 16:26:25.862659 | controller | - ' logger.go:42: 16:20:57 | watcher-tls/3-disable-podlevel-tls | + grep public' 2026-03-20 16:26:25.862663 | controller | - ' logger.go:42: 16:20:57 | watcher-tls/3-disable-podlevel-tls | ++ grep -c https' 2026-03-20 16:26:25.862667 | controller | - ' logger.go:42: 16:20:57 | watcher-tls/3-disable-podlevel-tls | + grep infra-optim' 2026-03-20 16:26:25.862671 | controller | - ' logger.go:42: 16:20:57 | watcher-tls/3-disable-podlevel-tls | + oc exec -n 2026-03-20 16:26:25.862676 | controller | watcher-kuttl-default openstackclient -- openstack endpoint list' 2026-03-20 16:26:25.862680 | controller | - ' logger.go:42: 16:20:59 | watcher-tls/3-disable-podlevel-tls | + ''['' 1 == 2026-03-20 16:26:25.862684 | controller | 1 '']''' 2026-03-20 16:26:25.862688 | controller | - ' logger.go:42: 16:20:59 | watcher-tls/3-disable-podlevel-tls | ++ oc get pods 2026-03-20 16:26:25.862708 | controller | -n openstack-operators -o name -l openstack.org/operator-name=watcher' 2026-03-20 16:26:25.862717 | controller | - ' logger.go:42: 16:21:00 | watcher-tls/3-disable-podlevel-tls | + ''['' '''' 2026-03-20 16:26:25.862721 | controller | == '''' '']''' 2026-03-20 16:26:25.862726 | controller | - ' logger.go:42: 16:21:00 | watcher-tls/3-disable-podlevel-tls | + exit 0' 2026-03-20 16:26:25.862730 | controller | - ' logger.go:42: 16:21:00 | watcher-tls/3-disable-podlevel-tls | test step completed 2026-03-20 16:26:25.862734 | controller | 3-disable-podlevel-tls' 2026-03-20 16:26:25.862738 | controller | - ' logger.go:42: 16:21:00 | watcher-tls/4-deploy-without-route | starting test 2026-03-20 16:26:25.862742 | controller | step 4-deploy-without-route' 2026-03-20 16:26:25.862747 | controller | - ' logger.go:42: 16:21:00 | watcher-tls/4-deploy-without-route | running command: 2026-03-20 16:26:25.862753 | controller | [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type=''json'' -p=''[{"op": 2026-03-20 16:26:25.862758 | controller | "replace", "path": "/spec/apiServiceTemplate/override", "value":{"service": { "internal": 2026-03-20 16:26:25.862764 | controller | {}, "public": { "metadata": { "annotations": { "metallb.universe.tf/address-pool": 2026-03-20 16:26:25.862769 | controller | "ctlplane", "metallb.universe.tf/allow-shared-ip": "ctlplane" } }, "spec": { "type": 2026-03-20 16:26:25.862775 | controller | "LoadBalancer" } } } }}]''' 2026-03-20 16:26:25.862780 | controller | - ' ]' 2026-03-20 16:26:25.862785 | controller | - ' logger.go:42: 16:21:00 | watcher-tls/4-deploy-without-route | watcher.watcher.openstack.org/watcher-kuttl 2026-03-20 16:26:25.862791 | controller | patched' 2026-03-20 16:26:25.862796 | controller | - ' logger.go:42: 16:21:00 | watcher-tls/4-deploy-without-route | running command: 2026-03-20 16:26:25.862802 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.862806 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-20 16:26:25.862810 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-20 16:26:25.862814 | controller | - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-20 16:26:25.862821 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-20 16:26:25.862826 | controller | - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.862830 | controller | == ${SERVICEID} ]' 2026-03-20 16:26:25.862834 | controller | - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-20 16:26:25.862838 | controller | ]' 2026-03-20 16:26:25.862842 | controller | - ' # If we are running the container locally, skip following test' 2026-03-20 16:26:25.862847 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-20 16:26:25.862851 | controller | == "" ]; then' 2026-03-20 16:26:25.862855 | controller | - ' exit 0' 2026-03-20 16:26:25.862859 | controller | - ' fi' 2026-03-20 16:26:25.862863 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-20 16:26:25.862867 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-20 16:26:25.862872 | controller | - ' counter=0' 2026-03-20 16:26:25.862878 | controller | - ' for i in ${env_variables}; do' 2026-03-20 16:26:25.862883 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-20 16:26:25.862889 | controller | - ' echo ${i}' 2026-03-20 16:26:25.862894 | controller | - ' counter=$((counter + 1))' 2026-03-20 16:26:25.862898 | controller | - ' fi' 2026-03-20 16:26:25.862903 | controller | - ' done' 2026-03-20 16:26:25.862909 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-20 16:26:25.862920 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-20 16:26:25.862924 | controller | - ' exit 1' 2026-03-20 16:26:25.862928 | controller | - ' else' 2026-03-20 16:26:25.862933 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-20 16:26:25.862937 | controller | - ' fi' 2026-03-20 16:26:25.862941 | controller | - ' ]' 2026-03-20 16:26:25.862946 | controller | - ' logger.go:42: 16:21:00 | watcher-tls/4-deploy-without-route | + oc exec -n 2026-03-20 16:26:25.862951 | controller | watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-20 16:26:25.862957 | controller | -c Type' 2026-03-20 16:26:25.862961 | controller | - ' logger.go:42: 16:21:00 | watcher-tls/4-deploy-without-route | ++ grep -c ''^watcher''' 2026-03-20 16:26:25.862975 | controller | - ' logger.go:42: 16:21:02 | watcher-tls/4-deploy-without-route | + ''['' 1 == 2026-03-20 16:26:25.862981 | controller | 1 '']''' 2026-03-20 16:26:25.862986 | controller | - ' logger.go:42: 16:21:02 | watcher-tls/4-deploy-without-route | ++ grep watcher' 2026-03-20 16:26:25.862990 | controller | - ' logger.go:42: 16:21:02 | watcher-tls/4-deploy-without-route | ++ awk ''{print 2026-03-20 16:26:25.862994 | controller | $1}''' 2026-03-20 16:26:25.862998 | controller | - ' logger.go:42: 16:21:02 | watcher-tls/4-deploy-without-route | ++ oc exec -n 2026-03-20 16:26:25.863003 | controller | watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-20 16:26:25.863009 | controller | -c Type -c ID' 2026-03-20 16:26:25.863014 | controller | - ' logger.go:42: 16:21:04 | watcher-tls/4-deploy-without-route | + SERVICEID=0bc8267151fd4fefbc0a3327396365a4' 2026-03-20 16:26:25.863018 | controller | - ' logger.go:42: 16:21:04 | watcher-tls/4-deploy-without-route | ++ oc get -n 2026-03-20 16:26:25.863022 | controller | watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-20 16:26:25.863026 | controller | - ' logger.go:42: 16:21:04 | watcher-tls/4-deploy-without-route | + ''['' 0bc8267151fd4fefbc0a3327396365a4 2026-03-20 16:26:25.863030 | controller | == 0bc8267151fd4fefbc0a3327396365a4 '']''' 2026-03-20 16:26:25.863034 | controller | - ' logger.go:42: 16:21:04 | watcher-tls/4-deploy-without-route | ++ oc get -n 2026-03-20 16:26:25.863039 | controller | watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-20 16:26:25.863043 | controller | - ' logger.go:42: 16:21:04 | watcher-tls/4-deploy-without-route | + ''['' -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.863050 | controller | '']''' 2026-03-20 16:26:25.863055 | controller | - ' logger.go:42: 16:21:04 | watcher-tls/4-deploy-without-route | ++ oc get pods 2026-03-20 16:26:25.863061 | controller | -n openstack-operators -o name -l openstack.org/operator-name=watcher' 2026-03-20 16:26:25.863066 | controller | - ' logger.go:42: 16:21:05 | watcher-tls/4-deploy-without-route | + ''['' '''' 2026-03-20 16:26:25.863072 | controller | == '''' '']''' 2026-03-20 16:26:25.863078 | controller | - ' logger.go:42: 16:21:05 | watcher-tls/4-deploy-without-route | + exit 0' 2026-03-20 16:26:25.863083 | controller | - ' logger.go:42: 16:21:06 | watcher-tls/4-deploy-without-route | running command: 2026-03-20 16:26:25.863089 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.863095 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-20 16:26:25.863101 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-20 16:26:25.863106 | controller | - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-20 16:26:25.863112 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-20 16:26:25.863117 | controller | - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.863123 | controller | == ${SERVICEID} ]' 2026-03-20 16:26:25.863128 | controller | - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-20 16:26:25.863133 | controller | ]' 2026-03-20 16:26:25.863139 | controller | - ' # If we are running the container locally, skip following test' 2026-03-20 16:26:25.863144 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-20 16:26:25.863150 | controller | == "" ]; then' 2026-03-20 16:26:25.863155 | controller | - ' exit 0' 2026-03-20 16:26:25.863161 | controller | - ' fi' 2026-03-20 16:26:25.863166 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-20 16:26:25.863172 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-20 16:26:25.863178 | controller | - ' counter=0' 2026-03-20 16:26:25.863183 | controller | - ' for i in ${env_variables}; do' 2026-03-20 16:26:25.863188 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-20 16:26:25.863194 | controller | - ' echo ${i}' 2026-03-20 16:26:25.863199 | controller | - ' counter=$((counter + 1))' 2026-03-20 16:26:25.863205 | controller | - ' fi' 2026-03-20 16:26:25.863209 | controller | - ' done' 2026-03-20 16:26:25.863214 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-20 16:26:25.863218 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-20 16:26:25.863222 | controller | - ' exit 1' 2026-03-20 16:26:25.863226 | controller | - ' else' 2026-03-20 16:26:25.863230 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-20 16:26:25.863235 | controller | - ' fi' 2026-03-20 16:26:25.863239 | controller | - ' ]' 2026-03-20 16:26:25.863243 | controller | - ' logger.go:42: 16:21:06 | watcher-tls/4-deploy-without-route | + oc exec -n 2026-03-20 16:26:25.863247 | controller | watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-20 16:26:25.863251 | controller | -c Type' 2026-03-20 16:26:25.863255 | controller | - ' logger.go:42: 16:21:06 | watcher-tls/4-deploy-without-route | ++ grep -c ''^watcher''' 2026-03-20 16:26:25.863260 | controller | - ' logger.go:42: 16:21:08 | watcher-tls/4-deploy-without-route | + ''['' 1 == 2026-03-20 16:26:25.863264 | controller | 1 '']''' 2026-03-20 16:26:25.863268 | controller | - ' logger.go:42: 16:21:08 | watcher-tls/4-deploy-without-route | ++ awk ''{print 2026-03-20 16:26:25.863272 | controller | $1}''' 2026-03-20 16:26:25.863276 | controller | - ' logger.go:42: 16:21:08 | watcher-tls/4-deploy-without-route | ++ oc exec -n 2026-03-20 16:26:25.863283 | controller | watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-20 16:26:25.863287 | controller | -c Type -c ID' 2026-03-20 16:26:25.863291 | controller | - ' logger.go:42: 16:21:08 | watcher-tls/4-deploy-without-route | ++ grep watcher' 2026-03-20 16:26:25.863305 | controller | - ' logger.go:42: 16:21:10 | watcher-tls/4-deploy-without-route | + SERVICEID=0bc8267151fd4fefbc0a3327396365a4' 2026-03-20 16:26:25.863310 | controller | - ' logger.go:42: 16:21:10 | watcher-tls/4-deploy-without-route | ++ oc get -n 2026-03-20 16:26:25.863314 | controller | watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-20 16:26:25.863318 | controller | - ' logger.go:42: 16:21:10 | watcher-tls/4-deploy-without-route | + ''['' 0bc8267151fd4fefbc0a3327396365a4 2026-03-20 16:26:25.863323 | controller | == 0bc8267151fd4fefbc0a3327396365a4 '']''' 2026-03-20 16:26:25.863327 | controller | - ' logger.go:42: 16:21:10 | watcher-tls/4-deploy-without-route | ++ oc get -n 2026-03-20 16:26:25.863331 | controller | watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-20 16:26:25.863335 | controller | - ' logger.go:42: 16:21:10 | watcher-tls/4-deploy-without-route | + ''['' -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.863339 | controller | '']''' 2026-03-20 16:26:25.863344 | controller | - ' logger.go:42: 16:21:10 | watcher-tls/4-deploy-without-route | ++ oc get pods 2026-03-20 16:26:25.863348 | controller | -n openstack-operators -o name -l openstack.org/operator-name=watcher' 2026-03-20 16:26:25.863352 | controller | - ' logger.go:42: 16:21:10 | watcher-tls/4-deploy-without-route | + ''['' '''' 2026-03-20 16:26:25.863356 | controller | == '''' '']''' 2026-03-20 16:26:25.863363 | controller | - ' logger.go:42: 16:21:10 | watcher-tls/4-deploy-without-route | + exit 0' 2026-03-20 16:26:25.863369 | controller | - ' logger.go:42: 16:21:10 | watcher-tls/4-deploy-without-route | test step completed 2026-03-20 16:26:25.863373 | controller | 4-deploy-without-route' 2026-03-20 16:26:25.863377 | controller | - ' logger.go:42: 16:21:10 | watcher-tls/5-disable-tls | starting test step 5-disable-tls' 2026-03-20 16:26:25.863381 | controller | - ' logger.go:42: 16:21:10 | watcher-tls/5-disable-tls | running command: [sh -c 2026-03-20 16:26:25.863385 | controller | oc patch watcher -n $NAMESPACE watcher-kuttl --type=''json'' -p=''[{"op": "replace", 2026-03-20 16:26:25.863389 | controller | "path": "/spec/apiServiceTemplate/override", "value":{}}]''' 2026-03-20 16:26:25.863394 | controller | - ' ]' 2026-03-20 16:26:25.863398 | controller | - ' logger.go:42: 16:21:10 | watcher-tls/5-disable-tls | watcher.watcher.openstack.org/watcher-kuttl 2026-03-20 16:26:25.863402 | controller | patched' 2026-03-20 16:26:25.863406 | controller | - ' logger.go:42: 16:21:10 | watcher-tls/5-disable-tls | running command: [sh -c 2026-03-20 16:26:25.863410 | controller | set -euxo pipefail' 2026-03-20 16:26:25.863415 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-20 16:26:25.863419 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-20 16:26:25.863425 | controller | - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-20 16:26:25.863429 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-20 16:26:25.863435 | controller | - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.863439 | controller | == ${SERVICEID} ]' 2026-03-20 16:26:25.863443 | controller | - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-20 16:26:25.863447 | controller | ]' 2026-03-20 16:26:25.863452 | controller | - ' # If we are running the container locally, skip following test' 2026-03-20 16:26:25.863456 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-20 16:26:25.863460 | controller | == "" ]; then' 2026-03-20 16:26:25.863464 | controller | - ' exit 0' 2026-03-20 16:26:25.863469 | controller | - ' fi' 2026-03-20 16:26:25.863475 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-20 16:26:25.863479 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-20 16:26:25.863483 | controller | - ' counter=0' 2026-03-20 16:26:25.863487 | controller | - ' for i in ${env_variables}; do' 2026-03-20 16:26:25.863492 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-20 16:26:25.863496 | controller | - ' echo ${i}' 2026-03-20 16:26:25.863500 | controller | - ' counter=$((counter + 1))' 2026-03-20 16:26:25.863504 | controller | - ' fi' 2026-03-20 16:26:25.863509 | controller | - ' done' 2026-03-20 16:26:25.863513 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-20 16:26:25.863517 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-20 16:26:25.863521 | controller | - ' exit 1' 2026-03-20 16:26:25.863525 | controller | - ' else' 2026-03-20 16:26:25.863530 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-20 16:26:25.863534 | controller | - ' fi' 2026-03-20 16:26:25.863538 | controller | - ' # check that no watcher endpoint uses https' 2026-03-20 16:26:25.863542 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-20 16:26:25.863546 | controller | infra-optim | [ $(grep -c https) == 0 ]' 2026-03-20 16:26:25.863551 | controller | - ' ]' 2026-03-20 16:26:25.863555 | controller | - ' logger.go:42: 16:21:10 | watcher-tls/5-disable-tls | + oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.863559 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-20 16:26:25.863563 | controller | - ' logger.go:42: 16:21:10 | watcher-tls/5-disable-tls | ++ grep -c ''^watcher''' 2026-03-20 16:26:25.863567 | controller | - ' logger.go:42: 16:21:12 | watcher-tls/5-disable-tls | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.863572 | controller | - ' logger.go:42: 16:21:12 | watcher-tls/5-disable-tls | ++ awk ''{print $1}''' 2026-03-20 16:26:25.863583 | controller | - ' logger.go:42: 16:21:12 | watcher-tls/5-disable-tls | ++ grep watcher' 2026-03-20 16:26:25.863589 | controller | - ' logger.go:42: 16:21:12 | watcher-tls/5-disable-tls | ++ oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.863593 | controller | openstackclient -- openstack service list -f value -c Name -c Type -c ID' 2026-03-20 16:26:25.863597 | controller | - ' logger.go:42: 16:21:14 | watcher-tls/5-disable-tls | + SERVICEID=0bc8267151fd4fefbc0a3327396365a4' 2026-03-20 16:26:25.863601 | controller | - ' logger.go:42: 16:21:14 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.863605 | controller | keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-20 16:26:25.863610 | controller | - ' logger.go:42: 16:21:15 | watcher-tls/5-disable-tls | + ''['' 0bc8267151fd4fefbc0a3327396365a4 2026-03-20 16:26:25.863614 | controller | == 0bc8267151fd4fefbc0a3327396365a4 '']''' 2026-03-20 16:26:25.863618 | controller | - ' logger.go:42: 16:21:15 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.863622 | controller | watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-20 16:26:25.863627 | controller | - ' logger.go:42: 16:21:15 | watcher-tls/5-disable-tls | + ''['' -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.863631 | controller | '']''' 2026-03-20 16:26:25.863635 | controller | - ' logger.go:42: 16:21:15 | watcher-tls/5-disable-tls | ++ oc get pods -n openstack-operators 2026-03-20 16:26:25.863640 | controller | -o name -l openstack.org/operator-name=watcher' 2026-03-20 16:26:25.863644 | controller | - ' logger.go:42: 16:21:15 | watcher-tls/5-disable-tls | + ''['' '''' == '''' '']''' 2026-03-20 16:26:25.863648 | controller | - ' logger.go:42: 16:21:15 | watcher-tls/5-disable-tls | + exit 0' 2026-03-20 16:26:25.863652 | controller | - ' logger.go:42: 16:21:16 | watcher-tls/5-disable-tls | running command: [sh -c 2026-03-20 16:26:25.863656 | controller | set -euxo pipefail' 2026-03-20 16:26:25.863664 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack service list -f value 2026-03-20 16:26:25.863668 | controller | -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-20 16:26:25.863672 | controller | - ' SERVICEID=$(oc exec -n ${NAMESPACE} openstackclient -- openstack service 2026-03-20 16:26:25.863677 | controller | list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-20 16:26:25.863681 | controller | - ' [ $(oc get -n ${NAMESPACE} keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.863685 | controller | == ${SERVICEID} ]' 2026-03-20 16:26:25.863689 | controller | - ' [ -n "$(oc get -n ${NAMESPACE} watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-20 16:26:25.863709 | controller | ]' 2026-03-20 16:26:25.863714 | controller | - ' # If we are running the container locally, skip following test' 2026-03-20 16:26:25.863718 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-20 16:26:25.863722 | controller | == "" ]; then' 2026-03-20 16:26:25.863726 | controller | - ' exit 0' 2026-03-20 16:26:25.863731 | controller | - ' fi' 2026-03-20 16:26:25.863735 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-20 16:26:25.863739 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-20 16:26:25.863743 | controller | - ' counter=0' 2026-03-20 16:26:25.863747 | controller | - ' for i in ${env_variables}; do' 2026-03-20 16:26:25.863755 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-20 16:26:25.863759 | controller | - ' echo ${i}' 2026-03-20 16:26:25.863763 | controller | - ' counter=$((counter + 1))' 2026-03-20 16:26:25.863767 | controller | - ' fi' 2026-03-20 16:26:25.863772 | controller | - ' done' 2026-03-20 16:26:25.863776 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-20 16:26:25.863806 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-20 16:26:25.863810 | controller | - ' exit 1' 2026-03-20 16:26:25.863814 | controller | - ' else' 2026-03-20 16:26:25.863819 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-20 16:26:25.863823 | controller | - ' fi' 2026-03-20 16:26:25.863827 | controller | - ' # check that no watcher endpoint uses https' 2026-03-20 16:26:25.863831 | controller | - ' oc exec -n ${NAMESPACE} openstackclient -- openstack endpoint list | grep 2026-03-20 16:26:25.863836 | controller | infra-optim | [ $(grep -c https) == 0 ]' 2026-03-20 16:26:25.863840 | controller | - ' ]' 2026-03-20 16:26:25.863844 | controller | - ' logger.go:42: 16:21:16 | watcher-tls/5-disable-tls | + oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.863848 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-20 16:26:25.863853 | controller | - ' logger.go:42: 16:21:16 | watcher-tls/5-disable-tls | ++ grep -c ''^watcher''' 2026-03-20 16:26:25.863857 | controller | - ' logger.go:42: 16:21:18 | watcher-tls/5-disable-tls | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.863861 | controller | - ' logger.go:42: 16:21:18 | watcher-tls/5-disable-tls | ++ grep watcher' 2026-03-20 16:26:25.863865 | controller | - ' logger.go:42: 16:21:18 | watcher-tls/5-disable-tls | ++ oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.863869 | controller | openstackclient -- openstack service list -f value -c Name -c Type -c ID' 2026-03-20 16:26:25.863874 | controller | - ' logger.go:42: 16:21:18 | watcher-tls/5-disable-tls | ++ awk ''{print $1}''' 2026-03-20 16:26:25.863878 | controller | - ' logger.go:42: 16:21:20 | watcher-tls/5-disable-tls | + SERVICEID=0bc8267151fd4fefbc0a3327396365a4' 2026-03-20 16:26:25.863882 | controller | - ' logger.go:42: 16:21:20 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.863894 | controller | keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-20 16:26:25.863898 | controller | - ' logger.go:42: 16:21:20 | watcher-tls/5-disable-tls | + ''['' 0bc8267151fd4fefbc0a3327396365a4 2026-03-20 16:26:25.863902 | controller | == 0bc8267151fd4fefbc0a3327396365a4 '']''' 2026-03-20 16:26:25.863909 | controller | - ' logger.go:42: 16:21:20 | watcher-tls/5-disable-tls | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.863913 | controller | watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-20 16:26:25.863918 | controller | - ' logger.go:42: 16:21:20 | watcher-tls/5-disable-tls | + ''['' -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.863922 | controller | '']''' 2026-03-20 16:26:25.863926 | controller | - ' logger.go:42: 16:21:20 | watcher-tls/5-disable-tls | ++ oc get pods -n openstack-operators 2026-03-20 16:26:25.863930 | controller | -o name -l openstack.org/operator-name=watcher' 2026-03-20 16:26:25.863935 | controller | - ' logger.go:42: 16:21:20 | watcher-tls/5-disable-tls | + ''['' '''' == '''' '']''' 2026-03-20 16:26:25.863939 | controller | - ' logger.go:42: 16:21:20 | watcher-tls/5-disable-tls | + exit 0' 2026-03-20 16:26:25.863943 | controller | - ' logger.go:42: 16:21:20 | watcher-tls/5-disable-tls | test step completed 5-disable-tls' 2026-03-20 16:26:25.863947 | controller | - ' logger.go:42: 16:21:20 | watcher-tls/6-cleanup-watcher | starting test step 2026-03-20 16:26:25.863951 | controller | 6-cleanup-watcher' 2026-03-20 16:26:25.863955 | controller | - ' logger.go:42: 16:21:20 | watcher-tls/6-cleanup-watcher | running command: [sh 2026-03-20 16:26:25.863960 | controller | -c set -ex' 2026-03-20 16:26:25.863964 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.863968 | controller | -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]' 2026-03-20 16:26:25.863972 | controller | - ' ]' 2026-03-20 16:26:25.863976 | controller | - ' logger.go:42: 16:21:20 | watcher-tls/6-cleanup-watcher | + oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.863981 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-20 16:26:25.863985 | controller | - ' logger.go:42: 16:21:20 | watcher-tls/6-cleanup-watcher | ++ grep -c ''^watcher''' 2026-03-20 16:26:25.863989 | controller | - ' logger.go:42: 16:21:23 | watcher-tls/6-cleanup-watcher | + ''['' 1 == 0 '']''' 2026-03-20 16:26:25.863993 | controller | - ' logger.go:42: 16:21:24 | watcher-tls/6-cleanup-watcher | running command: [sh 2026-03-20 16:26:25.863997 | controller | -c set -ex' 2026-03-20 16:26:25.864002 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.864006 | controller | -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]' 2026-03-20 16:26:25.864010 | controller | - ' ]' 2026-03-20 16:26:25.864014 | controller | - ' logger.go:42: 16:21:24 | watcher-tls/6-cleanup-watcher | + oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.864018 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-20 16:26:25.864023 | controller | - ' logger.go:42: 16:21:24 | watcher-tls/6-cleanup-watcher | ++ grep -c ''^watcher''' 2026-03-20 16:26:25.864027 | controller | - ' logger.go:42: 16:21:26 | watcher-tls/6-cleanup-watcher | + ''['' 0 == 0 '']''' 2026-03-20 16:26:25.864031 | controller | - ' logger.go:42: 16:21:27 | watcher-tls/6-cleanup-watcher | test step completed 2026-03-20 16:26:25.864035 | controller | 6-cleanup-watcher' 2026-03-20 16:26:25.864039 | controller | - ' logger.go:42: 16:21:27 | watcher-tls/7-cleanup-certs | starting test step 7-cleanup-certs' 2026-03-20 16:26:25.864044 | controller | - ' logger.go:42: 16:21:27 | watcher-tls/7-cleanup-certs | test step completed 2026-03-20 16:26:25.864048 | controller | 7-cleanup-certs' 2026-03-20 16:26:25.864054 | controller | - ' logger.go:42: 16:21:27 | watcher-tls | skipping kubernetes event logging' 2026-03-20 16:26:25.864058 | controller | - === CONT kuttl/harness/watcher-rmquser 2026-03-20 16:26:25.864062 | controller | - ' logger.go:42: 16:21:27 | watcher-rmquser | Skipping creation of user-supplied 2026-03-20 16:26:25.864067 | controller | namespace: watcher-kuttl-default' 2026-03-20 16:26:25.864071 | controller | - ' logger.go:42: 16:21:27 | watcher-rmquser/0-cleanup-watcher | starting test 2026-03-20 16:26:25.864075 | controller | step 0-cleanup-watcher' 2026-03-20 16:26:25.864081 | controller | - ' logger.go:42: 16:21:27 | watcher-rmquser/0-cleanup-watcher | test step completed 2026-03-20 16:26:25.864085 | controller | 0-cleanup-watcher' 2026-03-20 16:26:25.864089 | controller | - ' logger.go:42: 16:21:27 | watcher-rmquser/1-deploy | starting test step 1-deploy' 2026-03-20 16:26:25.864093 | controller | - ' logger.go:42: 16:21:27 | watcher-rmquser/1-deploy | Watcher:watcher-kuttl-default/watcher-kuttl 2026-03-20 16:26:25.864098 | controller | created' 2026-03-20 16:26:25.864102 | controller | - ' logger.go:42: 16:21:27 | watcher-rmquser/1-deploy | running command: [sh -c 2026-03-20 16:26:25.864106 | controller | set -euxo pipefail' 2026-03-20 16:26:25.864110 | controller | - ' ' 2026-03-20 16:26:25.864114 | controller | - ' # Wait for Watcher to be Ready' 2026-03-20 16:26:25.864119 | controller | - ' kubectl wait --for=condition=Ready watcher/watcher-kuttl -n $NAMESPACE 2026-03-20 16:26:25.864123 | controller | --timeout=300s' 2026-03-20 16:26:25.864127 | controller | - ' ' 2026-03-20 16:26:25.864131 | controller | - ' # Verify WatcherNotificationTransportURLReady condition exists and is True' 2026-03-20 16:26:25.864136 | controller | - ' kubectl get watcher watcher-kuttl -n $NAMESPACE -o jsonpath=''{.status.conditions[?(@.type=="WatcherNotificationTransportURLReady")].status}'' 2026-03-20 16:26:25.864140 | controller | | grep -q "True"' 2026-03-20 16:26:25.864144 | controller | - ' echo "WatcherNotificationTransportURLReady condition is True"' 2026-03-20 16:26:25.864148 | controller | - ' ' 2026-03-20 16:26:25.864159 | controller | - ' # Count TransportURL CRs - should be exactly 2 (one for messaging, one 2026-03-20 16:26:25.864164 | controller | for notifications)' 2026-03-20 16:26:25.864168 | controller | - ' transport_count=$(kubectl get transporturl -n $NAMESPACE -o name | grep 2026-03-20 16:26:25.864172 | controller | "watcher-kuttl-watcher-transport" | wc -l)' 2026-03-20 16:26:25.864176 | controller | - ' notification_transport_count=$(kubectl get transporturl -n $NAMESPACE -o 2026-03-20 16:26:25.864181 | controller | name | grep "watcher-kuttl-watcher-notification" | wc -l)' 2026-03-20 16:26:25.864185 | controller | - ' ' 2026-03-20 16:26:25.864189 | controller | - ' if [ "$transport_count" -ne "1" ]; then' 2026-03-20 16:26:25.864193 | controller | - ' echo "Expected 1 watcher-transport TransportURL, found $transport_count"' 2026-03-20 16:26:25.864198 | controller | - ' exit 1' 2026-03-20 16:26:25.864202 | controller | - ' fi' 2026-03-20 16:26:25.864206 | controller | - ' ' 2026-03-20 16:26:25.864210 | controller | - ' if [ "$notification_transport_count" -ne "1" ]; then' 2026-03-20 16:26:25.864214 | controller | - ' echo "Expected 1 notification-transport TransportURL, found $notification_transport_count"' 2026-03-20 16:26:25.864219 | controller | - ' exit 1' 2026-03-20 16:26:25.864223 | controller | - ' fi' 2026-03-20 16:26:25.864227 | controller | - ' ' 2026-03-20 16:26:25.864231 | controller | - ' echo "Correctly found 2 TransportURLs (separate clusters: transport and 2026-03-20 16:26:25.864236 | controller | notification)"' 2026-03-20 16:26:25.864240 | controller | - ' ' 2026-03-20 16:26:25.864244 | controller | - ' # Verify watcher-transport has correct user and vhost' 2026-03-20 16:26:25.864248 | controller | - ' transport_user=$(kubectl get transporturl watcher-kuttl-watcher-transport 2026-03-20 16:26:25.864252 | controller | -n $NAMESPACE -o jsonpath=''{.spec.username}'')' 2026-03-20 16:26:25.864257 | controller | - ' transport_vhost=$(kubectl get transporturl watcher-kuttl-watcher-transport 2026-03-20 16:26:25.864261 | controller | -n $NAMESPACE -o jsonpath=''{.spec.vhost}'')' 2026-03-20 16:26:25.864265 | controller | - ' if [ "$transport_user" != "watcher-rpc" ]; then' 2026-03-20 16:26:25.864269 | controller | - ' echo "Expected watcher-transport username ''watcher-rpc'', found ''$transport_user''"' 2026-03-20 16:26:25.864274 | controller | - ' exit 1' 2026-03-20 16:26:25.864278 | controller | - ' fi' 2026-03-20 16:26:25.864282 | controller | - ' if [ "$transport_vhost" != "watcher-rpc" ]; then' 2026-03-20 16:26:25.864286 | controller | - ' echo "Expected watcher-transport vhost ''watcher-rpc'', found ''$transport_vhost''"' 2026-03-20 16:26:25.864292 | controller | - ' exit 1' 2026-03-20 16:26:25.864296 | controller | - ' fi' 2026-03-20 16:26:25.864302 | controller | - ' echo "Watcher transport has correct user (watcher-rpc) and vhost (watcher-rpc)"' 2026-03-20 16:26:25.864307 | controller | - ' ' 2026-03-20 16:26:25.864311 | controller | - ' # Verify notification-transport has correct user and vhost' 2026-03-20 16:26:25.864315 | controller | - ' notif_user=$(kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications 2026-03-20 16:26:25.864319 | controller | -n $NAMESPACE -o jsonpath=''{.spec.username}'')' 2026-03-20 16:26:25.864323 | controller | - ' notif_vhost=$(kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications 2026-03-20 16:26:25.864328 | controller | -n $NAMESPACE -o jsonpath=''{.spec.vhost}'')' 2026-03-20 16:26:25.864332 | controller | - ' if [ "$notif_user" != "watcher-notifications" ]; then' 2026-03-20 16:26:25.864336 | controller | - ' echo "Expected notification-transport username ''watcher-notifications'', 2026-03-20 16:26:25.864340 | controller | found ''$notif_user''"' 2026-03-20 16:26:25.864344 | controller | - ' exit 1' 2026-03-20 16:26:25.864349 | controller | - ' fi' 2026-03-20 16:26:25.864353 | controller | - ' if [ "$notif_vhost" != "watcher-notifications" ]; then' 2026-03-20 16:26:25.864357 | controller | - ' echo "Expected notification-transport vhost ''watcher-notifications'', 2026-03-20 16:26:25.864361 | controller | found ''$notif_vhost''"' 2026-03-20 16:26:25.864365 | controller | - ' exit 1' 2026-03-20 16:26:25.864370 | controller | - ' fi' 2026-03-20 16:26:25.864374 | controller | - ' echo "Notification transport has correct user (watcher-notifications) and 2026-03-20 16:26:25.864378 | controller | vhost (watcher-notifications)"' 2026-03-20 16:26:25.864382 | controller | - ' ' 2026-03-20 16:26:25.864390 | controller | - ' # Verify that watcher.conf contains the notifications transport_url' 2026-03-20 16:26:25.864394 | controller | - ' WATCHER_API_POD=$(kubectl get pods -n $NAMESPACE -l "service=watcher-api" 2026-03-20 16:26:25.864398 | controller | -o custom-columns=:metadata.name --no-headers | grep -v ^$ | head -1)' 2026-03-20 16:26:25.864403 | controller | - ' if [ -z "${WATCHER_API_POD}" ]; then' 2026-03-20 16:26:25.864407 | controller | - ' echo "No watcher-api pod found"' 2026-03-20 16:26:25.864411 | controller | - ' exit 1' 2026-03-20 16:26:25.864415 | controller | - ' fi' 2026-03-20 16:26:25.864420 | controller | - ' # Verify RPC transport_url in DEFAULT section' 2026-03-20 16:26:25.864424 | controller | - ' rpc_transport_url=$(kubectl exec -n $NAMESPACE ${WATCHER_API_POD} -c watcher-api 2026-03-20 16:26:25.864428 | controller | -- cat /etc/watcher/watcher.conf.d/00-default.conf | grep -E ''^\[DEFAULT\]'' -A 2026-03-20 16:26:25.864435 | controller | 50 | grep ''transport_url'' | head -1 || true)' 2026-03-20 16:26:25.864439 | controller | - ' if [ -z "$rpc_transport_url" ]; then' 2026-03-20 16:26:25.864443 | controller | - ' echo "transport_url not found in DEFAULT section"' 2026-03-20 16:26:25.864448 | controller | - ' exit 1' 2026-03-20 16:26:25.864452 | controller | - ' fi' 2026-03-20 16:26:25.864456 | controller | - ' echo "Found RPC transport_url: $rpc_transport_url"' 2026-03-20 16:26:25.864460 | controller | - ' ' 2026-03-20 16:26:25.864465 | controller | - ' # Verify the RPC transport_url contains the correct vhost (watcher-rpc)' 2026-03-20 16:26:25.864469 | controller | - ' if ! echo "$rpc_transport_url" | grep -q ''/watcher-rpc''; then' 2026-03-20 16:26:25.864473 | controller | - ' echo "RPC transport_url does not contain expected vhost ''/watcher-rpc''"' 2026-03-20 16:26:25.864484 | controller | - ' exit 1' 2026-03-20 16:26:25.864489 | controller | - ' fi' 2026-03-20 16:26:25.864493 | controller | - ' echo "Successfully verified vhost ''watcher-rpc'' in RPC transport_url"' 2026-03-20 16:26:25.864497 | controller | - ' ' 2026-03-20 16:26:25.864502 | controller | - ' # Verify the RPC transport_url contains the correct username (watcher-rpc)' 2026-03-20 16:26:25.864506 | controller | - ' if ! echo "$rpc_transport_url" | grep -q ''watcher-rpc:''; then' 2026-03-20 16:26:25.864512 | controller | - ' echo "RPC transport_url does not contain expected username ''watcher-rpc:''"' 2026-03-20 16:26:25.864516 | controller | - ' exit 1' 2026-03-20 16:26:25.864520 | controller | - ' fi' 2026-03-20 16:26:25.864525 | controller | - ' echo "Successfully verified username ''watcher-rpc'' in RPC transport_url"' 2026-03-20 16:26:25.864529 | controller | - ' ' 2026-03-20 16:26:25.864533 | controller | - ' # Verify oslo_messaging_notifications section has transport_url configured' 2026-03-20 16:26:25.864537 | controller | - ' notif_transport_url=$(kubectl exec -n $NAMESPACE ${WATCHER_API_POD} -c 2026-03-20 16:26:25.864541 | controller | watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf | grep -A 5 ''\[oslo_messaging_notifications\]'' 2026-03-20 16:26:25.864546 | controller | | grep ''transport_url'' || true)' 2026-03-20 16:26:25.864550 | controller | - ' if [ -z "$notif_transport_url" ]; then' 2026-03-20 16:26:25.864556 | controller | - ' echo "transport_url not found in oslo_messaging_notifications section"' 2026-03-20 16:26:25.864560 | controller | - ' exit 1' 2026-03-20 16:26:25.864565 | controller | - ' fi' 2026-03-20 16:26:25.864569 | controller | - ' echo "Found notifications transport_url: $notif_transport_url"' 2026-03-20 16:26:25.864573 | controller | - ' ' 2026-03-20 16:26:25.864578 | controller | - ' # Verify the notifications transport_url contains the correct vhost (watcher-notifications)' 2026-03-20 16:26:25.864582 | controller | - ' if ! echo "$notif_transport_url" | grep -q ''/watcher-notifications''; 2026-03-20 16:26:25.864586 | controller | then' 2026-03-20 16:26:25.864590 | controller | - ' echo "Notifications transport_url does not contain expected vhost ''/watcher-notifications''"' 2026-03-20 16:26:25.864596 | controller | - ' exit 1' 2026-03-20 16:26:25.864600 | controller | - ' fi' 2026-03-20 16:26:25.864605 | controller | - ' echo "Successfully verified vhost ''watcher-notifications'' in notifications 2026-03-20 16:26:25.864609 | controller | transport_url"' 2026-03-20 16:26:25.864613 | controller | - ' ' 2026-03-20 16:26:25.864617 | controller | - ' # Verify the notifications transport_url contains the correct username 2026-03-20 16:26:25.864622 | controller | (watcher-notifications)' 2026-03-20 16:26:25.864626 | controller | - ' if ! echo "$notif_transport_url" | grep -q ''watcher-notifications:''; 2026-03-20 16:26:25.864630 | controller | then' 2026-03-20 16:26:25.864634 | controller | - ' echo "Notifications transport_url does not contain expected username 2026-03-20 16:26:25.864638 | controller | ''watcher-notifications:''"' 2026-03-20 16:26:25.864642 | controller | - ' exit 1' 2026-03-20 16:26:25.864647 | controller | - ' fi' 2026-03-20 16:26:25.864651 | controller | - ' echo "Successfully verified username ''watcher-notifications'' in notifications 2026-03-20 16:26:25.864655 | controller | transport_url"' 2026-03-20 16:26:25.864659 | controller | - ' ' 2026-03-20 16:26:25.864664 | controller | - ' exit 0' 2026-03-20 16:26:25.864668 | controller | - ' ]' 2026-03-20 16:26:25.864672 | controller | - ' logger.go:42: 16:21:27 | watcher-rmquser/1-deploy | + kubectl wait --for=condition=Ready 2026-03-20 16:26:25.864676 | controller | watcher/watcher-kuttl -n watcher-kuttl-default --timeout=300s' 2026-03-20 16:26:25.864681 | controller | - ' logger.go:42: 16:21:51 | watcher-rmquser/1-deploy | watcher.watcher.openstack.org/watcher-kuttl 2026-03-20 16:26:25.864685 | controller | condition met' 2026-03-20 16:26:25.864689 | controller | - ' logger.go:42: 16:21:51 | watcher-rmquser/1-deploy | + grep -q True' 2026-03-20 16:26:25.864709 | controller | - ' logger.go:42: 16:21:51 | watcher-rmquser/1-deploy | + kubectl get watcher watcher-kuttl 2026-03-20 16:26:25.864716 | controller | -n watcher-kuttl-default -o ''jsonpath={.status.conditions[?(@.type=="WatcherNotificationTransportURLReady")].status}''' 2026-03-20 16:26:25.864720 | controller | - ' logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | + echo ''WatcherNotificationTransportURLReady 2026-03-20 16:26:25.864724 | controller | condition is True''' 2026-03-20 16:26:25.864728 | controller | - ' logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | WatcherNotificationTransportURLReady 2026-03-20 16:26:25.864734 | controller | condition is True' 2026-03-20 16:26:25.864739 | controller | - ' logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | ++ grep watcher-kuttl-watcher-transport' 2026-03-20 16:26:25.864743 | controller | - ' logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | ++ kubectl get transporturl 2026-03-20 16:26:25.864747 | controller | -n watcher-kuttl-default -o name' 2026-03-20 16:26:25.864751 | controller | - ' logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | ++ wc -l' 2026-03-20 16:26:25.864755 | controller | - ' logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | + transport_count=1' 2026-03-20 16:26:25.864760 | controller | - ' logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | ++ kubectl get transporturl 2026-03-20 16:26:25.864764 | controller | -n watcher-kuttl-default -o name' 2026-03-20 16:26:25.864768 | controller | - ' logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | ++ wc -l' 2026-03-20 16:26:25.864772 | controller | - ' logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | ++ grep watcher-kuttl-watcher-notification' 2026-03-20 16:26:25.864776 | controller | - ' logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | + notification_transport_count=1' 2026-03-20 16:26:25.864781 | controller | - ' logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | + ''['' 1 -ne 1 '']''' 2026-03-20 16:26:25.864785 | controller | - ' logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | + ''['' 1 -ne 1 '']''' 2026-03-20 16:26:25.864796 | controller | - ' logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | + echo ''Correctly found 2026-03-20 16:26:25.864801 | controller | 2 TransportURLs (separate clusters: transport and notification)''' 2026-03-20 16:26:25.864805 | controller | - ' logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | Correctly found 2 TransportURLs 2026-03-20 16:26:25.864809 | controller | (separate clusters: transport and notification)' 2026-03-20 16:26:25.864813 | controller | - ' logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | ++ kubectl get transporturl 2026-03-20 16:26:25.864818 | controller | watcher-kuttl-watcher-transport -n watcher-kuttl-default -o ''jsonpath={.spec.username}''' 2026-03-20 16:26:25.864822 | controller | - ' logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | + transport_user=watcher-rpc' 2026-03-20 16:26:25.864826 | controller | - ' logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | ++ kubectl get transporturl 2026-03-20 16:26:25.864830 | controller | watcher-kuttl-watcher-transport -n watcher-kuttl-default -o ''jsonpath={.spec.vhost}''' 2026-03-20 16:26:25.864834 | controller | - ' logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | + transport_vhost=watcher-rpc' 2026-03-20 16:26:25.864838 | controller | - ' logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | + ''['' watcher-rpc ''!='' 2026-03-20 16:26:25.864843 | controller | watcher-rpc '']''' 2026-03-20 16:26:25.864847 | controller | - ' logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | + ''['' watcher-rpc ''!='' 2026-03-20 16:26:25.864851 | controller | watcher-rpc '']''' 2026-03-20 16:26:25.864855 | controller | - ' logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | + echo ''Watcher transport 2026-03-20 16:26:25.864859 | controller | has correct user (watcher-rpc) and vhost (watcher-rpc)''' 2026-03-20 16:26:25.864864 | controller | - ' logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | Watcher transport has correct 2026-03-20 16:26:25.864868 | controller | user (watcher-rpc) and vhost (watcher-rpc)' 2026-03-20 16:26:25.864872 | controller | - ' logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | ++ kubectl get transporturl 2026-03-20 16:26:25.864876 | controller | watcher-kuttl-watcher-notification-rabbitmq-notifications -n watcher-kuttl-default 2026-03-20 16:26:25.864880 | controller | -o ''jsonpath={.spec.username}''' 2026-03-20 16:26:25.864884 | controller | - ' logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | + notif_user=watcher-notifications' 2026-03-20 16:26:25.864889 | controller | - ' logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | ++ kubectl get transporturl 2026-03-20 16:26:25.864893 | controller | watcher-kuttl-watcher-notification-rabbitmq-notifications -n watcher-kuttl-default 2026-03-20 16:26:25.864897 | controller | -o ''jsonpath={.spec.vhost}''' 2026-03-20 16:26:25.864901 | controller | - ' logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | + notif_vhost=watcher-notifications' 2026-03-20 16:26:25.864907 | controller | - ' logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | + ''['' watcher-notifications 2026-03-20 16:26:25.864912 | controller | ''!='' watcher-notifications '']''' 2026-03-20 16:26:25.864916 | controller | - ' logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | + ''['' watcher-notifications 2026-03-20 16:26:25.864920 | controller | ''!='' watcher-notifications '']''' 2026-03-20 16:26:25.864924 | controller | - ' logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | + echo ''Notification transport 2026-03-20 16:26:25.864928 | controller | has correct user (watcher-notifications) and vhost (watcher-notifications)''' 2026-03-20 16:26:25.864935 | controller | - ' logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | Notification transport 2026-03-20 16:26:25.864939 | controller | has correct user (watcher-notifications) and vhost (watcher-notifications)' 2026-03-20 16:26:25.864943 | controller | - ' logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | ++ grep -v ''^$''' 2026-03-20 16:26:25.864948 | controller | - ' logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | ++ head -1' 2026-03-20 16:26:25.864952 | controller | - ' logger.go:42: 16:21:52 | watcher-rmquser/1-deploy | ++ kubectl get pods -n 2026-03-20 16:26:25.864956 | controller | watcher-kuttl-default -l service=watcher-api -o custom-columns=:metadata.name --no-headers' 2026-03-20 16:26:25.864960 | controller | - ' logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | + WATCHER_API_POD=watcher-kuttl-api-0' 2026-03-20 16:26:25.864964 | controller | - ' logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | + ''['' -z watcher-kuttl-api-0 2026-03-20 16:26:25.864969 | controller | '']''' 2026-03-20 16:26:25.864973 | controller | - ' logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | ++ grep -E ''^\[DEFAULT\]'' 2026-03-20 16:26:25.864977 | controller | -A 50' 2026-03-20 16:26:25.864981 | controller | - ' logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | ++ grep transport_url' 2026-03-20 16:26:25.864985 | controller | - ' logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | ++ kubectl exec -n watcher-kuttl-default 2026-03-20 16:26:25.864990 | controller | watcher-kuttl-api-0 -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.864994 | controller | - ' logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | ++ head -1' 2026-03-20 16:26:25.864998 | controller | - ' logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | + rpc_transport_url=''transport_url 2026-03-20 16:26:25.865002 | controller | = rabbit://watcher-rpc:3v320pCCv-k8XQA5ZC5ZgMG988bWL87e@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1''' 2026-03-20 16:26:25.865007 | controller | - ' logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | + ''['' -z ''transport_url 2026-03-20 16:26:25.865011 | controller | = rabbit://watcher-rpc:3v320pCCv-k8XQA5ZC5ZgMG988bWL87e@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'' 2026-03-20 16:26:25.865015 | controller | '']''' 2026-03-20 16:26:25.865026 | controller | - ' logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | + echo ''Found RPC transport_url: 2026-03-20 16:26:25.865032 | controller | transport_url = rabbit://watcher-rpc:3v320pCCv-k8XQA5ZC5ZgMG988bWL87e@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1''' 2026-03-20 16:26:25.865036 | controller | - ' logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | Found RPC transport_url: 2026-03-20 16:26:25.865041 | controller | transport_url = rabbit://watcher-rpc:3v320pCCv-k8XQA5ZC5ZgMG988bWL87e@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1' 2026-03-20 16:26:25.865045 | controller | - ' logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | + grep -q /watcher-rpc' 2026-03-20 16:26:25.865049 | controller | - ' logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | + echo ''transport_url 2026-03-20 16:26:25.865053 | controller | = rabbit://watcher-rpc:3v320pCCv-k8XQA5ZC5ZgMG988bWL87e@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1''' 2026-03-20 16:26:25.865058 | controller | - ' logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | + echo ''Successfully verified 2026-03-20 16:26:25.865062 | controller | vhost ''\''''watcher-rpc''\'''' in RPC transport_url''' 2026-03-20 16:26:25.865068 | controller | - ' logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | Successfully verified vhost 2026-03-20 16:26:25.865072 | controller | ''watcher-rpc'' in RPC transport_url' 2026-03-20 16:26:25.865077 | controller | - ' logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | + grep -q watcher-rpc:' 2026-03-20 16:26:25.865081 | controller | - ' logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | + echo ''transport_url 2026-03-20 16:26:25.865085 | controller | = rabbit://watcher-rpc:3v320pCCv-k8XQA5ZC5ZgMG988bWL87e@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1''' 2026-03-20 16:26:25.865089 | controller | - ' logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | + echo ''Successfully verified 2026-03-20 16:26:25.865093 | controller | username ''\''''watcher-rpc''\'''' in RPC transport_url''' 2026-03-20 16:26:25.865098 | controller | - ' logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | Successfully verified username 2026-03-20 16:26:25.865102 | controller | ''watcher-rpc'' in RPC transport_url' 2026-03-20 16:26:25.865106 | controller | - ' logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | ++ grep -A 5 ''\[oslo_messaging_notifications\]''' 2026-03-20 16:26:25.865110 | controller | - ' logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | ++ kubectl exec -n watcher-kuttl-default 2026-03-20 16:26:25.865115 | controller | watcher-kuttl-api-0 -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.865119 | controller | - ' logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | ++ grep transport_url' 2026-03-20 16:26:25.865123 | controller | - ' logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | + notif_transport_url=''transport_url 2026-03-20 16:26:25.865127 | controller | = rabbit://watcher-notifications:S1IIyezEXhXsAVBQzExoKD685nXbbjCW@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1''' 2026-03-20 16:26:25.865133 | controller | - ' logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | + ''['' -z ''transport_url 2026-03-20 16:26:25.865138 | controller | = rabbit://watcher-notifications:S1IIyezEXhXsAVBQzExoKD685nXbbjCW@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'' 2026-03-20 16:26:25.865142 | controller | '']''' 2026-03-20 16:26:25.865147 | controller | - ' logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | + echo ''Found notifications 2026-03-20 16:26:25.865151 | controller | transport_url: transport_url = rabbit://watcher-notifications:S1IIyezEXhXsAVBQzExoKD685nXbbjCW@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1''' 2026-03-20 16:26:25.865155 | controller | - ' logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | Found notifications transport_url: 2026-03-20 16:26:25.865159 | controller | transport_url = rabbit://watcher-notifications:S1IIyezEXhXsAVBQzExoKD685nXbbjCW@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1' 2026-03-20 16:26:25.865164 | controller | - ' logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | + grep -q /watcher-notifications' 2026-03-20 16:26:25.865168 | controller | - ' logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | + echo ''transport_url 2026-03-20 16:26:25.865172 | controller | = rabbit://watcher-notifications:S1IIyezEXhXsAVBQzExoKD685nXbbjCW@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1''' 2026-03-20 16:26:25.865176 | controller | - ' logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | + echo ''Successfully verified 2026-03-20 16:26:25.865181 | controller | vhost ''\''''watcher-notifications''\'''' in notifications transport_url''' 2026-03-20 16:26:25.865185 | controller | - ' logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | Successfully verified vhost 2026-03-20 16:26:25.865189 | controller | ''watcher-notifications'' in notifications transport_url' 2026-03-20 16:26:25.865193 | controller | - ' logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | + grep -q watcher-notifications:' 2026-03-20 16:26:25.865197 | controller | - ' logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | + echo ''transport_url 2026-03-20 16:26:25.865202 | controller | = rabbit://watcher-notifications:S1IIyezEXhXsAVBQzExoKD685nXbbjCW@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1''' 2026-03-20 16:26:25.865214 | controller | - ' logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | + echo ''Successfully verified 2026-03-20 16:26:25.865218 | controller | username ''\''''watcher-notifications''\'''' in notifications transport_url''' 2026-03-20 16:26:25.865223 | controller | - ' logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | Successfully verified username 2026-03-20 16:26:25.865227 | controller | ''watcher-notifications'' in notifications transport_url' 2026-03-20 16:26:25.865231 | controller | - ' logger.go:42: 16:21:53 | watcher-rmquser/1-deploy | + exit 0' 2026-03-20 16:26:25.865236 | controller | - ' logger.go:42: 16:21:54 | watcher-rmquser/1-deploy | running command: [sh -c 2026-03-20 16:26:25.865240 | controller | set -euxo pipefail' 2026-03-20 16:26:25.865244 | controller | - ' ' 2026-03-20 16:26:25.865248 | controller | - ' # Wait for Watcher to be Ready' 2026-03-20 16:26:25.865254 | controller | - ' kubectl wait --for=condition=Ready watcher/watcher-kuttl -n $NAMESPACE 2026-03-20 16:26:25.865259 | controller | --timeout=300s' 2026-03-20 16:26:25.865263 | controller | - ' ' 2026-03-20 16:26:25.865267 | controller | - ' # Verify WatcherNotificationTransportURLReady condition exists and is True' 2026-03-20 16:26:25.865271 | controller | - ' kubectl get watcher watcher-kuttl -n $NAMESPACE -o jsonpath=''{.status.conditions[?(@.type=="WatcherNotificationTransportURLReady")].status}'' 2026-03-20 16:26:25.865276 | controller | | grep -q "True"' 2026-03-20 16:26:25.865280 | controller | - ' echo "WatcherNotificationTransportURLReady condition is True"' 2026-03-20 16:26:25.865284 | controller | - ' ' 2026-03-20 16:26:25.865288 | controller | - ' # Count TransportURL CRs - should be exactly 2 (one for messaging, one 2026-03-20 16:26:25.865293 | controller | for notifications)' 2026-03-20 16:26:25.865297 | controller | - ' transport_count=$(kubectl get transporturl -n $NAMESPACE -o name | grep 2026-03-20 16:26:25.865301 | controller | "watcher-kuttl-watcher-transport" | wc -l)' 2026-03-20 16:26:25.865305 | controller | - ' notification_transport_count=$(kubectl get transporturl -n $NAMESPACE -o 2026-03-20 16:26:25.865309 | controller | name | grep "watcher-kuttl-watcher-notification" | wc -l)' 2026-03-20 16:26:25.865314 | controller | - ' ' 2026-03-20 16:26:25.865318 | controller | - ' if [ "$transport_count" -ne "1" ]; then' 2026-03-20 16:26:25.865322 | controller | - ' echo "Expected 1 watcher-transport TransportURL, found $transport_count"' 2026-03-20 16:26:25.865326 | controller | - ' exit 1' 2026-03-20 16:26:25.865331 | controller | - ' fi' 2026-03-20 16:26:25.865335 | controller | - ' ' 2026-03-20 16:26:25.865341 | controller | - ' if [ "$notification_transport_count" -ne "1" ]; then' 2026-03-20 16:26:25.865345 | controller | - ' echo "Expected 1 notification-transport TransportURL, found $notification_transport_count"' 2026-03-20 16:26:25.865349 | controller | - ' exit 1' 2026-03-20 16:26:25.865354 | controller | - ' fi' 2026-03-20 16:26:25.865358 | controller | - ' ' 2026-03-20 16:26:25.865362 | controller | - ' echo "Correctly found 2 TransportURLs (separate clusters: transport and 2026-03-20 16:26:25.865366 | controller | notification)"' 2026-03-20 16:26:25.865371 | controller | - ' ' 2026-03-20 16:26:25.865375 | controller | - ' # Verify watcher-transport has correct user and vhost' 2026-03-20 16:26:25.865379 | controller | - ' transport_user=$(kubectl get transporturl watcher-kuttl-watcher-transport 2026-03-20 16:26:25.865383 | controller | -n $NAMESPACE -o jsonpath=''{.spec.username}'')' 2026-03-20 16:26:25.865387 | controller | - ' transport_vhost=$(kubectl get transporturl watcher-kuttl-watcher-transport 2026-03-20 16:26:25.865392 | controller | -n $NAMESPACE -o jsonpath=''{.spec.vhost}'')' 2026-03-20 16:26:25.865396 | controller | - ' if [ "$transport_user" != "watcher-rpc" ]; then' 2026-03-20 16:26:25.865400 | controller | - ' echo "Expected watcher-transport username ''watcher-rpc'', found ''$transport_user''"' 2026-03-20 16:26:25.865404 | controller | - ' exit 1' 2026-03-20 16:26:25.865409 | controller | - ' fi' 2026-03-20 16:26:25.865413 | controller | - ' if [ "$transport_vhost" != "watcher-rpc" ]; then' 2026-03-20 16:26:25.865419 | controller | - ' echo "Expected watcher-transport vhost ''watcher-rpc'', found ''$transport_vhost''"' 2026-03-20 16:26:25.865423 | controller | - ' exit 1' 2026-03-20 16:26:25.865427 | controller | - ' fi' 2026-03-20 16:26:25.865435 | controller | - ' echo "Watcher transport has correct user (watcher-rpc) and vhost (watcher-rpc)"' 2026-03-20 16:26:25.865439 | controller | - ' ' 2026-03-20 16:26:25.865443 | controller | - ' # Verify notification-transport has correct user and vhost' 2026-03-20 16:26:25.865448 | controller | - ' notif_user=$(kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications 2026-03-20 16:26:25.865452 | controller | -n $NAMESPACE -o jsonpath=''{.spec.username}'')' 2026-03-20 16:26:25.865456 | controller | - ' notif_vhost=$(kubectl get transporturl watcher-kuttl-watcher-notification-rabbitmq-notifications 2026-03-20 16:26:25.865460 | controller | -n $NAMESPACE -o jsonpath=''{.spec.vhost}'')' 2026-03-20 16:26:25.865464 | controller | - ' if [ "$notif_user" != "watcher-notifications" ]; then' 2026-03-20 16:26:25.865469 | controller | - ' echo "Expected notification-transport username ''watcher-notifications'', 2026-03-20 16:26:25.865473 | controller | found ''$notif_user''"' 2026-03-20 16:26:25.865477 | controller | - ' exit 1' 2026-03-20 16:26:25.865481 | controller | - ' fi' 2026-03-20 16:26:25.865486 | controller | - ' if [ "$notif_vhost" != "watcher-notifications" ]; then' 2026-03-20 16:26:25.865490 | controller | - ' echo "Expected notification-transport vhost ''watcher-notifications'', 2026-03-20 16:26:25.865494 | controller | found ''$notif_vhost''"' 2026-03-20 16:26:25.865498 | controller | - ' exit 1' 2026-03-20 16:26:25.865503 | controller | - ' fi' 2026-03-20 16:26:25.865507 | controller | - ' echo "Notification transport has correct user (watcher-notifications) and 2026-03-20 16:26:25.865511 | controller | vhost (watcher-notifications)"' 2026-03-20 16:26:25.865515 | controller | - ' ' 2026-03-20 16:26:25.865520 | controller | - ' # Verify that watcher.conf contains the notifications transport_url' 2026-03-20 16:26:25.865530 | controller | - ' WATCHER_API_POD=$(kubectl get pods -n $NAMESPACE -l "service=watcher-api" 2026-03-20 16:26:25.865535 | controller | -o custom-columns=:metadata.name --no-headers | grep -v ^$ | head -1)' 2026-03-20 16:26:25.865539 | controller | - ' if [ -z "${WATCHER_API_POD}" ]; then' 2026-03-20 16:26:25.865543 | controller | - ' echo "No watcher-api pod found"' 2026-03-20 16:26:25.865547 | controller | - ' exit 1' 2026-03-20 16:26:25.865552 | controller | - ' fi' 2026-03-20 16:26:25.865556 | controller | - ' # Verify RPC transport_url in DEFAULT section' 2026-03-20 16:26:25.865560 | controller | - ' rpc_transport_url=$(kubectl exec -n $NAMESPACE ${WATCHER_API_POD} -c watcher-api 2026-03-20 16:26:25.865564 | controller | -- cat /etc/watcher/watcher.conf.d/00-default.conf | grep -E ''^\[DEFAULT\]'' -A 2026-03-20 16:26:25.865569 | controller | 50 | grep ''transport_url'' | head -1 || true)' 2026-03-20 16:26:25.865573 | controller | - ' if [ -z "$rpc_transport_url" ]; then' 2026-03-20 16:26:25.865577 | controller | - ' echo "transport_url not found in DEFAULT section"' 2026-03-20 16:26:25.865581 | controller | - ' exit 1' 2026-03-20 16:26:25.865586 | controller | - ' fi' 2026-03-20 16:26:25.865590 | controller | - ' echo "Found RPC transport_url: $rpc_transport_url"' 2026-03-20 16:26:25.865594 | controller | - ' ' 2026-03-20 16:26:25.865600 | controller | - ' # Verify the RPC transport_url contains the correct vhost (watcher-rpc)' 2026-03-20 16:26:25.865604 | controller | - ' if ! echo "$rpc_transport_url" | grep -q ''/watcher-rpc''; then' 2026-03-20 16:26:25.865609 | controller | - ' echo "RPC transport_url does not contain expected vhost ''/watcher-rpc''"' 2026-03-20 16:26:25.865613 | controller | - ' exit 1' 2026-03-20 16:26:25.865617 | controller | - ' fi' 2026-03-20 16:26:25.865621 | controller | - ' echo "Successfully verified vhost ''watcher-rpc'' in RPC transport_url"' 2026-03-20 16:26:25.865626 | controller | - ' ' 2026-03-20 16:26:25.865630 | controller | - ' # Verify the RPC transport_url contains the correct username (watcher-rpc)' 2026-03-20 16:26:25.865636 | controller | - ' if ! echo "$rpc_transport_url" | grep -q ''watcher-rpc:''; then' 2026-03-20 16:26:25.865640 | controller | - ' echo "RPC transport_url does not contain expected username ''watcher-rpc:''"' 2026-03-20 16:26:25.865644 | controller | - ' exit 1' 2026-03-20 16:26:25.865648 | controller | - ' fi' 2026-03-20 16:26:25.865653 | controller | - ' echo "Successfully verified username ''watcher-rpc'' in RPC transport_url"' 2026-03-20 16:26:25.865657 | controller | - ' ' 2026-03-20 16:26:25.865661 | controller | - ' # Verify oslo_messaging_notifications section has transport_url configured' 2026-03-20 16:26:25.865665 | controller | - ' notif_transport_url=$(kubectl exec -n $NAMESPACE ${WATCHER_API_POD} -c 2026-03-20 16:26:25.865670 | controller | watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf | grep -A 5 ''\[oslo_messaging_notifications\]'' 2026-03-20 16:26:25.865674 | controller | | grep ''transport_url'' || true)' 2026-03-20 16:26:25.865678 | controller | - ' if [ -z "$notif_transport_url" ]; then' 2026-03-20 16:26:25.865682 | controller | - ' echo "transport_url not found in oslo_messaging_notifications section"' 2026-03-20 16:26:25.865687 | controller | - ' exit 1' 2026-03-20 16:26:25.865691 | controller | - ' fi' 2026-03-20 16:26:25.865709 | controller | - ' echo "Found notifications transport_url: $notif_transport_url"' 2026-03-20 16:26:25.865713 | controller | - ' ' 2026-03-20 16:26:25.865717 | controller | - ' # Verify the notifications transport_url contains the correct vhost (watcher-notifications)' 2026-03-20 16:26:25.865722 | controller | - ' if ! echo "$notif_transport_url" | grep -q ''/watcher-notifications''; 2026-03-20 16:26:25.865726 | controller | then' 2026-03-20 16:26:25.865730 | controller | - ' echo "Notifications transport_url does not contain expected vhost ''/watcher-notifications''"' 2026-03-20 16:26:25.865734 | controller | - ' exit 1' 2026-03-20 16:26:25.865738 | controller | - ' fi' 2026-03-20 16:26:25.865743 | controller | - ' echo "Successfully verified vhost ''watcher-notifications'' in notifications 2026-03-20 16:26:25.865747 | controller | transport_url"' 2026-03-20 16:26:25.865751 | controller | - ' ' 2026-03-20 16:26:25.865755 | controller | - ' # Verify the notifications transport_url contains the correct username 2026-03-20 16:26:25.865760 | controller | (watcher-notifications)' 2026-03-20 16:26:25.865882 | controller | - ' if ! echo "$notif_transport_url" | grep -q ''watcher-notifications:''; 2026-03-20 16:26:25.865888 | controller | then' 2026-03-20 16:26:25.865892 | controller | - ' echo "Notifications transport_url does not contain expected username 2026-03-20 16:26:25.865897 | controller | ''watcher-notifications:''"' 2026-03-20 16:26:25.865901 | controller | - ' exit 1' 2026-03-20 16:26:25.865905 | controller | - ' fi' 2026-03-20 16:26:25.865912 | controller | - ' echo "Successfully verified username ''watcher-notifications'' in notifications 2026-03-20 16:26:25.865917 | controller | transport_url"' 2026-03-20 16:26:25.865921 | controller | - ' ' 2026-03-20 16:26:25.865925 | controller | - ' exit 0' 2026-03-20 16:26:25.865929 | controller | - ' ]' 2026-03-20 16:26:25.865934 | controller | - ' logger.go:42: 16:21:54 | watcher-rmquser/1-deploy | + kubectl wait --for=condition=Ready 2026-03-20 16:26:25.865938 | controller | watcher/watcher-kuttl -n watcher-kuttl-default --timeout=300s' 2026-03-20 16:26:25.865942 | controller | - ' logger.go:42: 16:21:54 | watcher-rmquser/1-deploy | watcher.watcher.openstack.org/watcher-kuttl 2026-03-20 16:26:25.865947 | controller | condition met' 2026-03-20 16:26:25.865951 | controller | - ' logger.go:42: 16:21:54 | watcher-rmquser/1-deploy | + kubectl get watcher watcher-kuttl 2026-03-20 16:26:25.865955 | controller | -n watcher-kuttl-default -o ''jsonpath={.status.conditions[?(@.type=="WatcherNotificationTransportURLReady")].status}''' 2026-03-20 16:26:25.865959 | controller | - ' logger.go:42: 16:21:54 | watcher-rmquser/1-deploy | + grep -q True' 2026-03-20 16:26:25.865964 | controller | - ' logger.go:42: 16:21:54 | watcher-rmquser/1-deploy | + echo ''WatcherNotificationTransportURLReady 2026-03-20 16:26:25.865968 | controller | condition is True''' 2026-03-20 16:26:25.865981 | controller | - ' logger.go:42: 16:21:54 | watcher-rmquser/1-deploy | WatcherNotificationTransportURLReady 2026-03-20 16:26:25.865986 | controller | condition is True' 2026-03-20 16:26:25.865990 | controller | - ' logger.go:42: 16:21:54 | watcher-rmquser/1-deploy | ++ grep watcher-kuttl-watcher-transport' 2026-03-20 16:26:25.865995 | controller | - ' logger.go:42: 16:21:54 | watcher-rmquser/1-deploy | ++ wc -l' 2026-03-20 16:26:25.865999 | controller | - ' logger.go:42: 16:21:54 | watcher-rmquser/1-deploy | ++ kubectl get transporturl 2026-03-20 16:26:25.866003 | controller | -n watcher-kuttl-default -o name' 2026-03-20 16:26:25.866008 | controller | - ' logger.go:42: 16:21:54 | watcher-rmquser/1-deploy | + transport_count=1' 2026-03-20 16:26:25.866012 | controller | - ' logger.go:42: 16:21:54 | watcher-rmquser/1-deploy | ++ grep watcher-kuttl-watcher-notification' 2026-03-20 16:26:25.866016 | controller | - ' logger.go:42: 16:21:54 | watcher-rmquser/1-deploy | ++ wc -l' 2026-03-20 16:26:25.866020 | controller | - ' logger.go:42: 16:21:54 | watcher-rmquser/1-deploy | ++ kubectl get transporturl 2026-03-20 16:26:25.866025 | controller | -n watcher-kuttl-default -o name' 2026-03-20 16:26:25.866029 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | + notification_transport_count=1' 2026-03-20 16:26:25.866033 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | + ''['' 1 -ne 1 '']''' 2026-03-20 16:26:25.866037 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | + ''['' 1 -ne 1 '']''' 2026-03-20 16:26:25.866041 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | + echo ''Correctly found 2026-03-20 16:26:25.866045 | controller | 2 TransportURLs (separate clusters: transport and notification)''' 2026-03-20 16:26:25.866050 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | Correctly found 2 TransportURLs 2026-03-20 16:26:25.866054 | controller | (separate clusters: transport and notification)' 2026-03-20 16:26:25.866058 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | ++ kubectl get transporturl 2026-03-20 16:26:25.866062 | controller | watcher-kuttl-watcher-transport -n watcher-kuttl-default -o ''jsonpath={.spec.username}''' 2026-03-20 16:26:25.866067 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | + transport_user=watcher-rpc' 2026-03-20 16:26:25.866071 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | ++ kubectl get transporturl 2026-03-20 16:26:25.866075 | controller | watcher-kuttl-watcher-transport -n watcher-kuttl-default -o ''jsonpath={.spec.vhost}''' 2026-03-20 16:26:25.866079 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | + transport_vhost=watcher-rpc' 2026-03-20 16:26:25.866083 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | + ''['' watcher-rpc ''!='' 2026-03-20 16:26:25.866088 | controller | watcher-rpc '']''' 2026-03-20 16:26:25.866092 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | + ''['' watcher-rpc ''!='' 2026-03-20 16:26:25.866096 | controller | watcher-rpc '']''' 2026-03-20 16:26:25.866100 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | + echo ''Watcher transport 2026-03-20 16:26:25.866104 | controller | has correct user (watcher-rpc) and vhost (watcher-rpc)''' 2026-03-20 16:26:25.866109 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | Watcher transport has correct 2026-03-20 16:26:25.866113 | controller | user (watcher-rpc) and vhost (watcher-rpc)' 2026-03-20 16:26:25.866117 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | ++ kubectl get transporturl 2026-03-20 16:26:25.866121 | controller | watcher-kuttl-watcher-notification-rabbitmq-notifications -n watcher-kuttl-default 2026-03-20 16:26:25.866126 | controller | -o ''jsonpath={.spec.username}''' 2026-03-20 16:26:25.866130 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | + notif_user=watcher-notifications' 2026-03-20 16:26:25.866134 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | ++ kubectl get transporturl 2026-03-20 16:26:25.866138 | controller | watcher-kuttl-watcher-notification-rabbitmq-notifications -n watcher-kuttl-default 2026-03-20 16:26:25.866144 | controller | -o ''jsonpath={.spec.vhost}''' 2026-03-20 16:26:25.866148 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | + notif_vhost=watcher-notifications' 2026-03-20 16:26:25.866152 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | + ''['' watcher-notifications 2026-03-20 16:26:25.866157 | controller | ''!='' watcher-notifications '']''' 2026-03-20 16:26:25.866161 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | + ''['' watcher-notifications 2026-03-20 16:26:25.866165 | controller | ''!='' watcher-notifications '']''' 2026-03-20 16:26:25.866169 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | + echo ''Notification transport 2026-03-20 16:26:25.866173 | controller | has correct user (watcher-notifications) and vhost (watcher-notifications)''' 2026-03-20 16:26:25.866178 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | Notification transport 2026-03-20 16:26:25.866182 | controller | has correct user (watcher-notifications) and vhost (watcher-notifications)' 2026-03-20 16:26:25.866186 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | ++ grep -v ''^$''' 2026-03-20 16:26:25.866190 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | ++ kubectl get pods -n 2026-03-20 16:26:25.866194 | controller | watcher-kuttl-default -l service=watcher-api -o custom-columns=:metadata.name --no-headers' 2026-03-20 16:26:25.866199 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | ++ head -1' 2026-03-20 16:26:25.866209 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | + WATCHER_API_POD=watcher-kuttl-api-0' 2026-03-20 16:26:25.866214 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | + ''['' -z watcher-kuttl-api-0 2026-03-20 16:26:25.866218 | controller | '']''' 2026-03-20 16:26:25.866222 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | ++ head -1' 2026-03-20 16:26:25.866227 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | ++ grep transport_url' 2026-03-20 16:26:25.866233 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | ++ grep -E ''^\[DEFAULT\]'' 2026-03-20 16:26:25.866237 | controller | -A 50' 2026-03-20 16:26:25.866241 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | ++ kubectl exec -n watcher-kuttl-default 2026-03-20 16:26:25.866245 | controller | watcher-kuttl-api-0 -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.866250 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | + rpc_transport_url=''transport_url 2026-03-20 16:26:25.866254 | controller | = rabbit://watcher-rpc:3v320pCCv-k8XQA5ZC5ZgMG988bWL87e@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1''' 2026-03-20 16:26:25.866258 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | + ''['' -z ''transport_url 2026-03-20 16:26:25.866262 | controller | = rabbit://watcher-rpc:3v320pCCv-k8XQA5ZC5ZgMG988bWL87e@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1'' 2026-03-20 16:26:25.866266 | controller | '']''' 2026-03-20 16:26:25.866270 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | + echo ''Found RPC transport_url: 2026-03-20 16:26:25.866275 | controller | transport_url = rabbit://watcher-rpc:3v320pCCv-k8XQA5ZC5ZgMG988bWL87e@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1''' 2026-03-20 16:26:25.866279 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | Found RPC transport_url: 2026-03-20 16:26:25.866283 | controller | transport_url = rabbit://watcher-rpc:3v320pCCv-k8XQA5ZC5ZgMG988bWL87e@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1' 2026-03-20 16:26:25.866287 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | + echo ''transport_url 2026-03-20 16:26:25.866291 | controller | = rabbit://watcher-rpc:3v320pCCv-k8XQA5ZC5ZgMG988bWL87e@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1''' 2026-03-20 16:26:25.866296 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | + grep -q /watcher-rpc' 2026-03-20 16:26:25.866300 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | + echo ''Successfully verified 2026-03-20 16:26:25.866305 | controller | vhost ''\''''watcher-rpc''\'''' in RPC transport_url''' 2026-03-20 16:26:25.866310 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | Successfully verified vhost 2026-03-20 16:26:25.866314 | controller | ''watcher-rpc'' in RPC transport_url' 2026-03-20 16:26:25.866318 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | + grep -q watcher-rpc:' 2026-03-20 16:26:25.866322 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | + echo ''transport_url 2026-03-20 16:26:25.866327 | controller | = rabbit://watcher-rpc:3v320pCCv-k8XQA5ZC5ZgMG988bWL87e@rabbitmq.watcher-kuttl-default.svc:5671/watcher-rpc?ssl=1''' 2026-03-20 16:26:25.866331 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | + echo ''Successfully verified 2026-03-20 16:26:25.866335 | controller | username ''\''''watcher-rpc''\'''' in RPC transport_url''' 2026-03-20 16:26:25.866339 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | Successfully verified username 2026-03-20 16:26:25.866343 | controller | ''watcher-rpc'' in RPC transport_url' 2026-03-20 16:26:25.866348 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | ++ grep -A 5 ''\[oslo_messaging_notifications\]''' 2026-03-20 16:26:25.866352 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | ++ grep transport_url' 2026-03-20 16:26:25.866356 | controller | - ' logger.go:42: 16:21:55 | watcher-rmquser/1-deploy | ++ kubectl exec -n watcher-kuttl-default 2026-03-20 16:26:25.866360 | controller | watcher-kuttl-api-0 -c watcher-api -- cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.866364 | controller | - ' logger.go:42: 16:21:56 | watcher-rmquser/1-deploy | + notif_transport_url=''transport_url 2026-03-20 16:26:25.866368 | controller | = rabbit://watcher-notifications:S1IIyezEXhXsAVBQzExoKD685nXbbjCW@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1''' 2026-03-20 16:26:25.866373 | controller | - ' logger.go:42: 16:21:56 | watcher-rmquser/1-deploy | + ''['' -z ''transport_url 2026-03-20 16:26:25.866377 | controller | = rabbit://watcher-notifications:S1IIyezEXhXsAVBQzExoKD685nXbbjCW@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1'' 2026-03-20 16:26:25.866381 | controller | '']''' 2026-03-20 16:26:25.866385 | controller | - ' logger.go:42: 16:21:56 | watcher-rmquser/1-deploy | + echo ''Found notifications 2026-03-20 16:26:25.866390 | controller | transport_url: transport_url = rabbit://watcher-notifications:S1IIyezEXhXsAVBQzExoKD685nXbbjCW@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1''' 2026-03-20 16:26:25.866394 | controller | - ' logger.go:42: 16:21:56 | watcher-rmquser/1-deploy | Found notifications transport_url: 2026-03-20 16:26:25.866404 | controller | transport_url = rabbit://watcher-notifications:S1IIyezEXhXsAVBQzExoKD685nXbbjCW@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1' 2026-03-20 16:26:25.866409 | controller | - ' logger.go:42: 16:21:56 | watcher-rmquser/1-deploy | + grep -q /watcher-notifications' 2026-03-20 16:26:25.866413 | controller | - ' logger.go:42: 16:21:56 | watcher-rmquser/1-deploy | + echo ''transport_url 2026-03-20 16:26:25.866417 | controller | = rabbit://watcher-notifications:S1IIyezEXhXsAVBQzExoKD685nXbbjCW@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1''' 2026-03-20 16:26:25.866421 | controller | - ' logger.go:42: 16:21:56 | watcher-rmquser/1-deploy | + echo ''Successfully verified 2026-03-20 16:26:25.866426 | controller | vhost ''\''''watcher-notifications''\'''' in notifications transport_url''' 2026-03-20 16:26:25.866430 | controller | - ' logger.go:42: 16:21:56 | watcher-rmquser/1-deploy | Successfully verified vhost 2026-03-20 16:26:25.866434 | controller | ''watcher-notifications'' in notifications transport_url' 2026-03-20 16:26:25.866438 | controller | - ' logger.go:42: 16:21:56 | watcher-rmquser/1-deploy | + grep -q watcher-notifications:' 2026-03-20 16:26:25.866443 | controller | - ' logger.go:42: 16:21:56 | watcher-rmquser/1-deploy | + echo ''transport_url 2026-03-20 16:26:25.866447 | controller | = rabbit://watcher-notifications:S1IIyezEXhXsAVBQzExoKD685nXbbjCW@rabbitmq-notifications.watcher-kuttl-default.svc:5671/watcher-notifications?ssl=1''' 2026-03-20 16:26:25.866453 | controller | - ' logger.go:42: 16:21:56 | watcher-rmquser/1-deploy | + echo ''Successfully verified 2026-03-20 16:26:25.866457 | controller | username ''\''''watcher-notifications''\'''' in notifications transport_url''' 2026-03-20 16:26:25.866462 | controller | - ' logger.go:42: 16:21:56 | watcher-rmquser/1-deploy | Successfully verified username 2026-03-20 16:26:25.866466 | controller | ''watcher-notifications'' in notifications transport_url' 2026-03-20 16:26:25.866470 | controller | - ' logger.go:42: 16:21:56 | watcher-rmquser/1-deploy | + exit 0' 2026-03-20 16:26:25.866474 | controller | - ' logger.go:42: 16:21:56 | watcher-rmquser/1-deploy | test step completed 1-deploy' 2026-03-20 16:26:25.866478 | controller | - ' logger.go:42: 16:21:56 | watcher-rmquser/2-cleanup-watcher | starting test 2026-03-20 16:26:25.866483 | controller | step 2-cleanup-watcher' 2026-03-20 16:26:25.866487 | controller | - ' logger.go:42: 16:22:01 | watcher-rmquser/2-cleanup-watcher | test step completed 2026-03-20 16:26:25.866491 | controller | 2-cleanup-watcher' 2026-03-20 16:26:25.866495 | controller | - ' logger.go:42: 16:22:01 | watcher-rmquser | skipping kubernetes event logging' 2026-03-20 16:26:25.866499 | controller | - === CONT kuttl/harness/watcher 2026-03-20 16:26:25.866504 | controller | - ' logger.go:42: 16:22:01 | watcher | Skipping creation of user-supplied namespace: 2026-03-20 16:26:25.866508 | controller | watcher-kuttl-default' 2026-03-20 16:26:25.866512 | controller | - ' logger.go:42: 16:22:01 | watcher/0-cleanup-watcher | starting test step 0-cleanup-watcher' 2026-03-20 16:26:25.866517 | controller | - ' logger.go:42: 16:22:01 | watcher/0-cleanup-watcher | test step completed 0-cleanup-watcher' 2026-03-20 16:26:25.866521 | controller | - ' logger.go:42: 16:22:01 | watcher/1-deploy-with-defaults | starting test step 2026-03-20 16:26:25.866525 | controller | 1-deploy-with-defaults' 2026-03-20 16:26:25.866529 | controller | - ' logger.go:42: 16:22:02 | watcher/1-deploy-with-defaults | Watcher:watcher-kuttl-default/watcher-kuttl 2026-03-20 16:26:25.866533 | controller | created' 2026-03-20 16:26:25.866538 | controller | - ' logger.go:42: 16:22:02 | watcher/1-deploy-with-defaults | running command: 2026-03-20 16:26:25.866542 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.866549 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.866553 | controller | -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-20 16:26:25.866557 | controller | - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack 2026-03-20 16:26:25.866561 | controller | service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-20 16:26:25.866566 | controller | - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.866581 | controller | == ${SERVICEID} ]' 2026-03-20 16:26:25.866585 | controller | - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-20 16:26:25.866590 | controller | ]' 2026-03-20 16:26:25.866594 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.866598 | controller | -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]' 2026-03-20 16:26:25.866604 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.866609 | controller | -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" 2026-03-20 16:26:25.866613 | controller | == 2 ]' 2026-03-20 16:26:25.866617 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.866621 | controller | -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''region_name = regionOne'')" 2026-03-20 16:26:25.866625 | controller | -ge 1 ]' 2026-03-20 16:26:25.866630 | controller | - ' # If we are running the container locally, skip following test' 2026-03-20 16:26:25.866635 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-20 16:26:25.866640 | controller | == "" ]; then' 2026-03-20 16:26:25.866644 | controller | - ' exit 0' 2026-03-20 16:26:25.866648 | controller | - ' fi' 2026-03-20 16:26:25.866659 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-20 16:26:25.866663 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-20 16:26:25.866667 | controller | - ' counter=0' 2026-03-20 16:26:25.866672 | controller | - ' for i in ${env_variables}; do' 2026-03-20 16:26:25.866676 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-20 16:26:25.866680 | controller | - ' echo ${i}' 2026-03-20 16:26:25.866684 | controller | - ' counter=$((counter + 1))' 2026-03-20 16:26:25.866689 | controller | - ' fi' 2026-03-20 16:26:25.866707 | controller | - ' done' 2026-03-20 16:26:25.866714 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-20 16:26:25.866718 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-20 16:26:25.866722 | controller | - ' exit 1' 2026-03-20 16:26:25.866727 | controller | - ' else' 2026-03-20 16:26:25.866731 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-20 16:26:25.866735 | controller | - ' fi' 2026-03-20 16:26:25.866739 | controller | - ' ]' 2026-03-20 16:26:25.866743 | controller | - ' logger.go:42: 16:22:02 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.866748 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-20 16:26:25.866752 | controller | - ' logger.go:42: 16:22:02 | watcher/1-deploy-with-defaults | ++ grep -c ''^watcher''' 2026-03-20 16:26:25.866756 | controller | - ' logger.go:42: 16:22:04 | watcher/1-deploy-with-defaults | + ''['' 0 == 1 '']''' 2026-03-20 16:26:25.866760 | controller | - ' logger.go:42: 16:22:05 | watcher/1-deploy-with-defaults | running command: 2026-03-20 16:26:25.866764 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.866768 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.866773 | controller | -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-20 16:26:25.866777 | controller | - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack 2026-03-20 16:26:25.866781 | controller | service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-20 16:26:25.866785 | controller | - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.866789 | controller | == ${SERVICEID} ]' 2026-03-20 16:26:25.866793 | controller | - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-20 16:26:25.866798 | controller | ]' 2026-03-20 16:26:25.866802 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.866806 | controller | -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]' 2026-03-20 16:26:25.866810 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.866814 | controller | -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" 2026-03-20 16:26:25.866819 | controller | == 2 ]' 2026-03-20 16:26:25.866823 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.866827 | controller | -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''region_name = regionOne'')" 2026-03-20 16:26:25.866831 | controller | -ge 1 ]' 2026-03-20 16:26:25.866835 | controller | - ' # If we are running the container locally, skip following test' 2026-03-20 16:26:25.866839 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-20 16:26:25.866844 | controller | == "" ]; then' 2026-03-20 16:26:25.866848 | controller | - ' exit 0' 2026-03-20 16:26:25.866854 | controller | - ' fi' 2026-03-20 16:26:25.866858 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-20 16:26:25.866863 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-20 16:26:25.866867 | controller | - ' counter=0' 2026-03-20 16:26:25.866871 | controller | - ' for i in ${env_variables}; do' 2026-03-20 16:26:25.866875 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-20 16:26:25.866880 | controller | - ' echo ${i}' 2026-03-20 16:26:25.866884 | controller | - ' counter=$((counter + 1))' 2026-03-20 16:26:25.866888 | controller | - ' fi' 2026-03-20 16:26:25.866892 | controller | - ' done' 2026-03-20 16:26:25.866896 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-20 16:26:25.866901 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-20 16:26:25.866905 | controller | - ' exit 1' 2026-03-20 16:26:25.866909 | controller | - ' else' 2026-03-20 16:26:25.866913 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-20 16:26:25.866917 | controller | - ' fi' 2026-03-20 16:26:25.866922 | controller | - ' ]' 2026-03-20 16:26:25.866926 | controller | - ' logger.go:42: 16:22:05 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.866932 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-20 16:26:25.866936 | controller | - ' logger.go:42: 16:22:05 | watcher/1-deploy-with-defaults | ++ grep -c ''^watcher''' 2026-03-20 16:26:25.866940 | controller | - ' logger.go:42: 16:22:07 | watcher/1-deploy-with-defaults | + ''['' 0 == 1 '']''' 2026-03-20 16:26:25.866945 | controller | - ' logger.go:42: 16:22:08 | watcher/1-deploy-with-defaults | running command: 2026-03-20 16:26:25.866949 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.866953 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.866957 | controller | -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-20 16:26:25.866962 | controller | - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack 2026-03-20 16:26:25.866973 | controller | service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-20 16:26:25.866977 | controller | - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.866981 | controller | == ${SERVICEID} ]' 2026-03-20 16:26:25.866986 | controller | - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-20 16:26:25.866990 | controller | ]' 2026-03-20 16:26:25.866994 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.866998 | controller | -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]' 2026-03-20 16:26:25.867002 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.867006 | controller | -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" 2026-03-20 16:26:25.867011 | controller | == 2 ]' 2026-03-20 16:26:25.867015 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.867019 | controller | -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''region_name = regionOne'')" 2026-03-20 16:26:25.867023 | controller | -ge 1 ]' 2026-03-20 16:26:25.867028 | controller | - ' # If we are running the container locally, skip following test' 2026-03-20 16:26:25.867032 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-20 16:26:25.867036 | controller | == "" ]; then' 2026-03-20 16:26:25.867040 | controller | - ' exit 0' 2026-03-20 16:26:25.867044 | controller | - ' fi' 2026-03-20 16:26:25.867050 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-20 16:26:25.867054 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-20 16:26:25.867061 | controller | - ' counter=0' 2026-03-20 16:26:25.867066 | controller | - ' for i in ${env_variables}; do' 2026-03-20 16:26:25.867070 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-20 16:26:25.867074 | controller | - ' echo ${i}' 2026-03-20 16:26:25.867080 | controller | - ' counter=$((counter + 1))' 2026-03-20 16:26:25.867084 | controller | - ' fi' 2026-03-20 16:26:25.867089 | controller | - ' done' 2026-03-20 16:26:25.867093 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-20 16:26:25.867097 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-20 16:26:25.867102 | controller | - ' exit 1' 2026-03-20 16:26:25.867106 | controller | - ' else' 2026-03-20 16:26:25.867110 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-20 16:26:25.867114 | controller | - ' fi' 2026-03-20 16:26:25.867119 | controller | - ' ]' 2026-03-20 16:26:25.867123 | controller | - ' logger.go:42: 16:22:08 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.867127 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-20 16:26:25.867132 | controller | - ' logger.go:42: 16:22:08 | watcher/1-deploy-with-defaults | ++ grep -c ''^watcher''' 2026-03-20 16:26:25.867137 | controller | - ' logger.go:42: 16:22:10 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.867142 | controller | - ' logger.go:42: 16:22:10 | watcher/1-deploy-with-defaults | ++ oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.867146 | controller | openstackclient -- openstack service list -f value -c Name -c Type -c ID' 2026-03-20 16:26:25.867150 | controller | - ' logger.go:42: 16:22:10 | watcher/1-deploy-with-defaults | ++ awk ''{print $1}''' 2026-03-20 16:26:25.867154 | controller | - ' logger.go:42: 16:22:10 | watcher/1-deploy-with-defaults | ++ grep watcher' 2026-03-20 16:26:25.867159 | controller | - ' logger.go:42: 16:22:12 | watcher/1-deploy-with-defaults | + SERVICEID=50cd2d8870524a79924faff92882dc56' 2026-03-20 16:26:25.867163 | controller | - ' logger.go:42: 16:22:12 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.867167 | controller | keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-20 16:26:25.867171 | controller | - ' logger.go:42: 16:22:13 | watcher/1-deploy-with-defaults | + ''['' 50cd2d8870524a79924faff92882dc56 2026-03-20 16:26:25.867175 | controller | == 50cd2d8870524a79924faff92882dc56 '']''' 2026-03-20 16:26:25.867180 | controller | - ' logger.go:42: 16:22:13 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.867184 | controller | watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-20 16:26:25.867188 | controller | - ' logger.go:42: 16:22:13 | watcher/1-deploy-with-defaults | + ''['' -n '''' '']''' 2026-03-20 16:26:25.867192 | controller | - ' logger.go:42: 16:22:14 | watcher/1-deploy-with-defaults | running command: 2026-03-20 16:26:25.867196 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.867201 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.867205 | controller | -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-20 16:26:25.867209 | controller | - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack 2026-03-20 16:26:25.867213 | controller | service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-20 16:26:25.867218 | controller | - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.867222 | controller | == ${SERVICEID} ]' 2026-03-20 16:26:25.867228 | controller | - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-20 16:26:25.867232 | controller | ]' 2026-03-20 16:26:25.867237 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.867247 | controller | -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]' 2026-03-20 16:26:25.867254 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.867258 | controller | -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" 2026-03-20 16:26:25.867262 | controller | == 2 ]' 2026-03-20 16:26:25.867267 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.867271 | controller | -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''region_name = regionOne'')" 2026-03-20 16:26:25.867275 | controller | -ge 1 ]' 2026-03-20 16:26:25.867279 | controller | - ' # If we are running the container locally, skip following test' 2026-03-20 16:26:25.867284 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-20 16:26:25.867288 | controller | == "" ]; then' 2026-03-20 16:26:25.867292 | controller | - ' exit 0' 2026-03-20 16:26:25.867296 | controller | - ' fi' 2026-03-20 16:26:25.867301 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-20 16:26:25.867305 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-20 16:26:25.867309 | controller | - ' counter=0' 2026-03-20 16:26:25.867313 | controller | - ' for i in ${env_variables}; do' 2026-03-20 16:26:25.867318 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-20 16:26:25.867322 | controller | - ' echo ${i}' 2026-03-20 16:26:25.867326 | controller | - ' counter=$((counter + 1))' 2026-03-20 16:26:25.867330 | controller | - ' fi' 2026-03-20 16:26:25.867334 | controller | - ' done' 2026-03-20 16:26:25.867339 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-20 16:26:25.867343 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-20 16:26:25.867347 | controller | - ' exit 1' 2026-03-20 16:26:25.867351 | controller | - ' else' 2026-03-20 16:26:25.867355 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-20 16:26:25.867360 | controller | - ' fi' 2026-03-20 16:26:25.867364 | controller | - ' ]' 2026-03-20 16:26:25.867368 | controller | - ' logger.go:42: 16:22:14 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.867372 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-20 16:26:25.867377 | controller | - ' logger.go:42: 16:22:14 | watcher/1-deploy-with-defaults | ++ grep -c ''^watcher''' 2026-03-20 16:26:25.867381 | controller | - ' logger.go:42: 16:22:16 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.867385 | controller | - ' logger.go:42: 16:22:16 | watcher/1-deploy-with-defaults | ++ oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.867389 | controller | openstackclient -- openstack service list -f value -c Name -c Type -c ID' 2026-03-20 16:26:25.867393 | controller | - ' logger.go:42: 16:22:16 | watcher/1-deploy-with-defaults | ++ grep watcher' 2026-03-20 16:26:25.867398 | controller | - ' logger.go:42: 16:22:16 | watcher/1-deploy-with-defaults | ++ awk ''{print $1}''' 2026-03-20 16:26:25.867402 | controller | - ' logger.go:42: 16:22:18 | watcher/1-deploy-with-defaults | + SERVICEID=50cd2d8870524a79924faff92882dc56' 2026-03-20 16:26:25.867406 | controller | - ' logger.go:42: 16:22:18 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.867410 | controller | keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-20 16:26:25.867415 | controller | - ' logger.go:42: 16:22:19 | watcher/1-deploy-with-defaults | + ''['' 50cd2d8870524a79924faff92882dc56 2026-03-20 16:26:25.867419 | controller | == 50cd2d8870524a79924faff92882dc56 '']''' 2026-03-20 16:26:25.867423 | controller | - ' logger.go:42: 16:22:19 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.867427 | controller | watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-20 16:26:25.867431 | controller | - ' logger.go:42: 16:22:19 | watcher/1-deploy-with-defaults | + ''['' -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.867438 | controller | '']''' 2026-03-20 16:26:25.867442 | controller | - ' logger.go:42: 16:22:19 | watcher/1-deploy-with-defaults | ++ base64 -d' 2026-03-20 16:26:25.867446 | controller | - ' logger.go:42: 16:22:19 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.867450 | controller | secret watcher-kuttl-api-config-data -o ''jsonpath={.data.my\.cnf}''' 2026-03-20 16:26:25.867454 | controller | - ' logger.go:42: 16:22:19 | watcher/1-deploy-with-defaults | ++ grep -c ssl=1' 2026-03-20 16:26:25.867459 | controller | - ' logger.go:42: 16:22:19 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.867463 | controller | - ' logger.go:42: 16:22:19 | watcher/1-deploy-with-defaults | ++ base64 -d' 2026-03-20 16:26:25.867467 | controller | - ' logger.go:42: 16:22:19 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.867471 | controller | secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}''' 2026-03-20 16:26:25.867475 | controller | - ' logger.go:42: 16:22:19 | watcher/1-deploy-with-defaults | ++ grep -c ''cafile 2026-03-20 16:26:25.867480 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem''' 2026-03-20 16:26:25.867484 | controller | - ' logger.go:42: 16:22:19 | watcher/1-deploy-with-defaults | + ''['' 2 == 2 '']''' 2026-03-20 16:26:25.867488 | controller | - ' logger.go:42: 16:22:19 | watcher/1-deploy-with-defaults | ++ grep -c ''region_name 2026-03-20 16:26:25.867492 | controller | = regionOne''' 2026-03-20 16:26:25.867503 | controller | - ' logger.go:42: 16:22:19 | watcher/1-deploy-with-defaults | ++ base64 -d' 2026-03-20 16:26:25.867507 | controller | - ' logger.go:42: 16:22:19 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.867512 | controller | secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}''' 2026-03-20 16:26:25.867516 | controller | - ' logger.go:42: 16:22:19 | watcher/1-deploy-with-defaults | + ''['' 9 -ge 1 '']''' 2026-03-20 16:26:25.867520 | controller | - ' logger.go:42: 16:22:19 | watcher/1-deploy-with-defaults | ++ oc get pods -n 2026-03-20 16:26:25.867524 | controller | openstack-operators -o name -l openstack.org/operator-name=watcher' 2026-03-20 16:26:25.867531 | controller | - ' logger.go:42: 16:22:19 | watcher/1-deploy-with-defaults | + ''['' '''' == '''' 2026-03-20 16:26:25.867535 | controller | '']''' 2026-03-20 16:26:25.867539 | controller | - ' logger.go:42: 16:22:19 | watcher/1-deploy-with-defaults | + exit 0' 2026-03-20 16:26:25.867543 | controller | - ' logger.go:42: 16:22:20 | watcher/1-deploy-with-defaults | running command: 2026-03-20 16:26:25.867547 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.867552 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.867556 | controller | -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-20 16:26:25.867560 | controller | - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack 2026-03-20 16:26:25.867564 | controller | service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-20 16:26:25.867568 | controller | - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.867573 | controller | == ${SERVICEID} ]' 2026-03-20 16:26:25.867577 | controller | - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-20 16:26:25.867581 | controller | ]' 2026-03-20 16:26:25.867587 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.867591 | controller | -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]' 2026-03-20 16:26:25.867596 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.867600 | controller | -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" 2026-03-20 16:26:25.867614 | controller | == 2 ]' 2026-03-20 16:26:25.867619 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.867623 | controller | -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''region_name = regionOne'')" 2026-03-20 16:26:25.867627 | controller | -ge 1 ]' 2026-03-20 16:26:25.867631 | controller | - ' # If we are running the container locally, skip following test' 2026-03-20 16:26:25.867635 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-20 16:26:25.867641 | controller | == "" ]; then' 2026-03-20 16:26:25.867646 | controller | - ' exit 0' 2026-03-20 16:26:25.867650 | controller | - ' fi' 2026-03-20 16:26:25.867654 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-20 16:26:25.867658 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-20 16:26:25.867663 | controller | - ' counter=0' 2026-03-20 16:26:25.867667 | controller | - ' for i in ${env_variables}; do' 2026-03-20 16:26:25.867671 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-20 16:26:25.867675 | controller | - ' echo ${i}' 2026-03-20 16:26:25.867680 | controller | - ' counter=$((counter + 1))' 2026-03-20 16:26:25.867684 | controller | - ' fi' 2026-03-20 16:26:25.867688 | controller | - ' done' 2026-03-20 16:26:25.867703 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-20 16:26:25.867709 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-20 16:26:25.867713 | controller | - ' exit 1' 2026-03-20 16:26:25.867718 | controller | - ' else' 2026-03-20 16:26:25.867722 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-20 16:26:25.867726 | controller | - ' fi' 2026-03-20 16:26:25.867730 | controller | - ' ]' 2026-03-20 16:26:25.867734 | controller | - ' logger.go:42: 16:22:20 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.867738 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-20 16:26:25.867743 | controller | - ' logger.go:42: 16:22:20 | watcher/1-deploy-with-defaults | ++ grep -c ''^watcher''' 2026-03-20 16:26:25.867747 | controller | - ' logger.go:42: 16:22:22 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.867751 | controller | - ' logger.go:42: 16:22:22 | watcher/1-deploy-with-defaults | ++ grep watcher' 2026-03-20 16:26:25.867755 | controller | - ' logger.go:42: 16:22:22 | watcher/1-deploy-with-defaults | ++ oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.867759 | controller | openstackclient -- openstack service list -f value -c Name -c Type -c ID' 2026-03-20 16:26:25.867764 | controller | - ' logger.go:42: 16:22:22 | watcher/1-deploy-with-defaults | ++ awk ''{print $1}''' 2026-03-20 16:26:25.867768 | controller | - ' logger.go:42: 16:22:25 | watcher/1-deploy-with-defaults | + SERVICEID=50cd2d8870524a79924faff92882dc56' 2026-03-20 16:26:25.867772 | controller | - ' logger.go:42: 16:22:25 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.867776 | controller | keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-20 16:26:25.867780 | controller | - ' logger.go:42: 16:22:25 | watcher/1-deploy-with-defaults | + ''['' 50cd2d8870524a79924faff92882dc56 2026-03-20 16:26:25.867809 | controller | == 50cd2d8870524a79924faff92882dc56 '']''' 2026-03-20 16:26:25.867813 | controller | - ' logger.go:42: 16:22:25 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.867820 | controller | watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-20 16:26:25.867824 | controller | - ' logger.go:42: 16:22:25 | watcher/1-deploy-with-defaults | + ''['' -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.867829 | controller | '']''' 2026-03-20 16:26:25.867833 | controller | - ' logger.go:42: 16:22:25 | watcher/1-deploy-with-defaults | ++ grep -c ssl=1' 2026-03-20 16:26:25.867837 | controller | - ' logger.go:42: 16:22:25 | watcher/1-deploy-with-defaults | ++ base64 -d' 2026-03-20 16:26:25.867853 | controller | - ' logger.go:42: 16:22:25 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.867858 | controller | secret watcher-kuttl-api-config-data -o ''jsonpath={.data.my\.cnf}''' 2026-03-20 16:26:25.867862 | controller | - ' logger.go:42: 16:22:25 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.867866 | controller | - ' logger.go:42: 16:22:25 | watcher/1-deploy-with-defaults | ++ base64 -d' 2026-03-20 16:26:25.867870 | controller | - ' logger.go:42: 16:22:25 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.867874 | controller | secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}''' 2026-03-20 16:26:25.867878 | controller | - ' logger.go:42: 16:22:25 | watcher/1-deploy-with-defaults | ++ grep -c ''cafile 2026-03-20 16:26:25.867883 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem''' 2026-03-20 16:26:25.867887 | controller | - ' logger.go:42: 16:22:25 | watcher/1-deploy-with-defaults | + ''['' 2 == 2 '']''' 2026-03-20 16:26:25.867891 | controller | - ' logger.go:42: 16:22:25 | watcher/1-deploy-with-defaults | ++ base64 -d' 2026-03-20 16:26:25.867895 | controller | - ' logger.go:42: 16:22:25 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.867899 | controller | secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}''' 2026-03-20 16:26:25.867904 | controller | - ' logger.go:42: 16:22:25 | watcher/1-deploy-with-defaults | ++ grep -c ''region_name 2026-03-20 16:26:25.867908 | controller | = regionOne''' 2026-03-20 16:26:25.867912 | controller | - ' logger.go:42: 16:22:25 | watcher/1-deploy-with-defaults | + ''['' 9 -ge 1 '']''' 2026-03-20 16:26:25.867916 | controller | - ' logger.go:42: 16:22:25 | watcher/1-deploy-with-defaults | ++ oc get pods -n 2026-03-20 16:26:25.867921 | controller | openstack-operators -o name -l openstack.org/operator-name=watcher' 2026-03-20 16:26:25.867925 | controller | - ' logger.go:42: 16:22:26 | watcher/1-deploy-with-defaults | + ''['' '''' == '''' 2026-03-20 16:26:25.867929 | controller | '']''' 2026-03-20 16:26:25.867933 | controller | - ' logger.go:42: 16:22:26 | watcher/1-deploy-with-defaults | + exit 0' 2026-03-20 16:26:25.867937 | controller | - ' logger.go:42: 16:22:27 | watcher/1-deploy-with-defaults | running command: 2026-03-20 16:26:25.867941 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.867946 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.867950 | controller | -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-20 16:26:25.867954 | controller | - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack 2026-03-20 16:26:25.867958 | controller | service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-20 16:26:25.867962 | controller | - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.867968 | controller | == ${SERVICEID} ]' 2026-03-20 16:26:25.867973 | controller | - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-20 16:26:25.867977 | controller | ]' 2026-03-20 16:26:25.867981 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.867985 | controller | -o jsonpath=''{.data.my\.cnf}''|base64 -d|grep -c ''ssl=1'')" == 1 ]' 2026-03-20 16:26:25.867989 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.867994 | controller | -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem'')" 2026-03-20 16:26:25.867998 | controller | == 2 ]' 2026-03-20 16:26:25.868002 | controller | - ' [ "$(oc get -n watcher-kuttl-default secret watcher-kuttl-api-config-data 2026-03-20 16:26:25.868006 | controller | -o jsonpath=''{.data.00-default\.conf}''|base64 -d|grep -c ''region_name = regionOne'')" 2026-03-20 16:26:25.868010 | controller | -ge 1 ]' 2026-03-20 16:26:25.868016 | controller | - ' # If we are running the container locally, skip following test' 2026-03-20 16:26:25.868020 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-20 16:26:25.868024 | controller | == "" ]; then' 2026-03-20 16:26:25.868028 | controller | - ' exit 0' 2026-03-20 16:26:25.868033 | controller | - ' fi' 2026-03-20 16:26:25.868037 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-20 16:26:25.868043 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-20 16:26:25.868047 | controller | - ' counter=0' 2026-03-20 16:26:25.868051 | controller | - ' for i in ${env_variables}; do' 2026-03-20 16:26:25.868055 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-20 16:26:25.868067 | controller | - ' echo ${i}' 2026-03-20 16:26:25.868071 | controller | - ' counter=$((counter + 1))' 2026-03-20 16:26:25.868075 | controller | - ' fi' 2026-03-20 16:26:25.868080 | controller | - ' done' 2026-03-20 16:26:25.868084 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-20 16:26:25.868088 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-20 16:26:25.868092 | controller | - ' exit 1' 2026-03-20 16:26:25.868096 | controller | - ' else' 2026-03-20 16:26:25.868101 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-20 16:26:25.868105 | controller | - ' fi' 2026-03-20 16:26:25.868109 | controller | - ' ]' 2026-03-20 16:26:25.868113 | controller | - ' logger.go:42: 16:22:27 | watcher/1-deploy-with-defaults | + oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.868117 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-20 16:26:25.868122 | controller | - ' logger.go:42: 16:22:27 | watcher/1-deploy-with-defaults | ++ grep -c ''^watcher''' 2026-03-20 16:26:25.868126 | controller | - ' logger.go:42: 16:22:29 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.868130 | controller | - ' logger.go:42: 16:22:29 | watcher/1-deploy-with-defaults | ++ oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.868134 | controller | openstackclient -- openstack service list -f value -c Name -c Type -c ID' 2026-03-20 16:26:25.868138 | controller | - ' logger.go:42: 16:22:29 | watcher/1-deploy-with-defaults | ++ grep watcher' 2026-03-20 16:26:25.868142 | controller | - ' logger.go:42: 16:22:29 | watcher/1-deploy-with-defaults | ++ awk ''{print $1}''' 2026-03-20 16:26:25.868147 | controller | - ' logger.go:42: 16:22:31 | watcher/1-deploy-with-defaults | + SERVICEID=50cd2d8870524a79924faff92882dc56' 2026-03-20 16:26:25.868151 | controller | - ' logger.go:42: 16:22:31 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.868155 | controller | keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-20 16:26:25.868159 | controller | - ' logger.go:42: 16:22:31 | watcher/1-deploy-with-defaults | + ''['' 50cd2d8870524a79924faff92882dc56 2026-03-20 16:26:25.868163 | controller | == 50cd2d8870524a79924faff92882dc56 '']''' 2026-03-20 16:26:25.868167 | controller | - ' logger.go:42: 16:22:31 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.868172 | controller | watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-20 16:26:25.868176 | controller | - ' logger.go:42: 16:22:31 | watcher/1-deploy-with-defaults | + ''['' -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.868180 | controller | '']''' 2026-03-20 16:26:25.868187 | controller | - ' logger.go:42: 16:22:31 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.868191 | controller | secret watcher-kuttl-api-config-data -o ''jsonpath={.data.my\.cnf}''' 2026-03-20 16:26:25.868195 | controller | - ' logger.go:42: 16:22:31 | watcher/1-deploy-with-defaults | ++ grep -c ssl=1' 2026-03-20 16:26:25.868200 | controller | - ' logger.go:42: 16:22:31 | watcher/1-deploy-with-defaults | ++ base64 -d' 2026-03-20 16:26:25.868204 | controller | - ' logger.go:42: 16:22:31 | watcher/1-deploy-with-defaults | + ''['' 1 == 1 '']''' 2026-03-20 16:26:25.868210 | controller | - ' logger.go:42: 16:22:31 | watcher/1-deploy-with-defaults | ++ base64 -d' 2026-03-20 16:26:25.868214 | controller | - ' logger.go:42: 16:22:31 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.868218 | controller | secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}''' 2026-03-20 16:26:25.868223 | controller | - ' logger.go:42: 16:22:31 | watcher/1-deploy-with-defaults | ++ grep -c ''cafile 2026-03-20 16:26:25.868227 | controller | = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem''' 2026-03-20 16:26:25.868231 | controller | - ' logger.go:42: 16:22:31 | watcher/1-deploy-with-defaults | + ''['' 2 == 2 '']''' 2026-03-20 16:26:25.868235 | controller | - ' logger.go:42: 16:22:31 | watcher/1-deploy-with-defaults | ++ base64 -d' 2026-03-20 16:26:25.868239 | controller | - ' logger.go:42: 16:22:31 | watcher/1-deploy-with-defaults | ++ oc get -n watcher-kuttl-default 2026-03-20 16:26:25.868244 | controller | secret watcher-kuttl-api-config-data -o ''jsonpath={.data.00-default\.conf}''' 2026-03-20 16:26:25.868248 | controller | - ' logger.go:42: 16:22:31 | watcher/1-deploy-with-defaults | ++ grep -c ''region_name 2026-03-20 16:26:25.868252 | controller | = regionOne''' 2026-03-20 16:26:25.868256 | controller | - ' logger.go:42: 16:22:32 | watcher/1-deploy-with-defaults | + ''['' 9 -ge 1 '']''' 2026-03-20 16:26:25.868260 | controller | - ' logger.go:42: 16:22:32 | watcher/1-deploy-with-defaults | ++ oc get pods -n 2026-03-20 16:26:25.868264 | controller | openstack-operators -o name -l openstack.org/operator-name=watcher' 2026-03-20 16:26:25.868269 | controller | - ' logger.go:42: 16:22:32 | watcher/1-deploy-with-defaults | + ''['' '''' == '''' 2026-03-20 16:26:25.868273 | controller | '']''' 2026-03-20 16:26:25.868277 | controller | - ' logger.go:42: 16:22:32 | watcher/1-deploy-with-defaults | + exit 0' 2026-03-20 16:26:25.868281 | controller | - ' logger.go:42: 16:22:32 | watcher/1-deploy-with-defaults | test step completed 2026-03-20 16:26:25.868285 | controller | 1-deploy-with-defaults' 2026-03-20 16:26:25.868290 | controller | - ' logger.go:42: 16:22:32 | watcher/2-cleanup-watcher | starting test step 2-cleanup-watcher' 2026-03-20 16:26:25.868294 | controller | - ' logger.go:42: 16:22:32 | watcher/2-cleanup-watcher | test step completed 2-cleanup-watcher' 2026-03-20 16:26:25.868305 | controller | - ' logger.go:42: 16:22:32 | watcher/3-precreate-mariadbaccount | starting test 2026-03-20 16:26:25.868309 | controller | step 3-precreate-mariadbaccount' 2026-03-20 16:26:25.868313 | controller | - ' logger.go:42: 16:22:32 | watcher/3-precreate-mariadbaccount | MariaDBAccount:watcher-kuttl-default/watcher-precreated 2026-03-20 16:26:25.868318 | controller | created' 2026-03-20 16:26:25.868322 | controller | - ' logger.go:42: 16:22:32 | watcher/3-precreate-mariadbaccount | test step completed 2026-03-20 16:26:25.868326 | controller | 3-precreate-mariadbaccount' 2026-03-20 16:26:25.868330 | controller | - ' logger.go:42: 16:22:32 | watcher/4-deploy-with-precreated-account | starting 2026-03-20 16:26:25.868334 | controller | test step 4-deploy-with-precreated-account' 2026-03-20 16:26:25.868339 | controller | - ' logger.go:42: 16:22:32 | watcher/4-deploy-with-precreated-account | Secret:watcher-kuttl-default/custom-prometheus-config 2026-03-20 16:26:25.868343 | controller | created' 2026-03-20 16:26:25.868347 | controller | - ' logger.go:42: 16:22:32 | watcher/4-deploy-with-precreated-account | Watcher:watcher-kuttl-default/watcher-kuttl 2026-03-20 16:26:25.868352 | controller | created' 2026-03-20 16:26:25.868356 | controller | - ' logger.go:42: 16:22:32 | watcher/4-deploy-with-precreated-account | running 2026-03-20 16:26:25.868360 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.868365 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.868369 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.868376 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.868380 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.868385 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.868390 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-20 16:26:25.868396 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-20 16:26:25.868400 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-20 16:26:25.868404 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.868409 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-20 16:26:25.868413 | controller | == 1 ]' 2026-03-20 16:26:25.868417 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.868421 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-20 16:26:25.868426 | controller | - ' else' 2026-03-20 16:26:25.868430 | controller | - ' exit 1' 2026-03-20 16:26:25.868434 | controller | - ' fi' 2026-03-20 16:26:25.868438 | controller | - ' ]' 2026-03-20 16:26:25.868442 | controller | - ' logger.go:42: 16:22:32 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.868447 | controller | watcher-kuttl-default' 2026-03-20 16:26:25.868453 | controller | - ' logger.go:42: 16:22:32 | watcher/4-deploy-with-precreated-account | Already 2026-03-20 16:26:25.868457 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.868461 | controller | - ' logger.go:42: 16:22:32 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.868465 | controller | -v ''^$''' 2026-03-20 16:26:25.868470 | controller | - ' logger.go:42: 16:22:32 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-20 16:26:25.868474 | controller | -1' 2026-03-20 16:26:25.868478 | controller | - ' logger.go:42: 16:22:32 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-20 16:26:25.868482 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.868486 | controller | - ' logger.go:42: 16:22:32 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' 2026-03-20 16:26:25.868490 | controller | - ' logger.go:42: 16:22:32 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.868495 | controller | -n watcher-kuttl-api-0 '']''' 2026-03-20 16:26:25.868499 | controller | - ' logger.go:42: 16:22:32 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.868503 | controller | -c ''^# Global config''' 2026-03-20 16:26:25.868507 | controller | - ' logger.go:42: 16:22:32 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-20 16:26:25.868511 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' 2026-03-20 16:26:25.868515 | controller | - ' logger.go:42: 16:22:34 | watcher/4-deploy-with-precreated-account | error: 2026-03-20 16:26:25.868520 | controller | Internal error occurred: error executing command in container: container is not 2026-03-20 16:26:25.868524 | controller | created or running' 2026-03-20 16:26:25.868528 | controller | - ' logger.go:42: 16:22:34 | watcher/4-deploy-with-precreated-account | ++ echo' 2026-03-20 16:26:25.868532 | controller | - ' logger.go:42: 16:22:34 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.868536 | controller | 0 == 1 '']''' 2026-03-20 16:26:25.868540 | controller | - ' logger.go:42: 16:22:35 | watcher/4-deploy-with-precreated-account | running 2026-03-20 16:26:25.868544 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.868549 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.868553 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.868557 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.868561 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.868565 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.868585 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-20 16:26:25.868590 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-20 16:26:25.868594 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-20 16:26:25.868598 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.868602 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-20 16:26:25.868607 | controller | == 1 ]' 2026-03-20 16:26:25.868611 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.868615 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-20 16:26:25.868619 | controller | - ' else' 2026-03-20 16:26:25.868624 | controller | - ' exit 1' 2026-03-20 16:26:25.868628 | controller | - ' fi' 2026-03-20 16:26:25.868632 | controller | - ' ]' 2026-03-20 16:26:25.868636 | controller | - ' logger.go:42: 16:22:35 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.868640 | controller | watcher-kuttl-default' 2026-03-20 16:26:25.868645 | controller | - ' logger.go:42: 16:22:35 | watcher/4-deploy-with-precreated-account | Already 2026-03-20 16:26:25.868649 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.868653 | controller | - ' logger.go:42: 16:22:35 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-20 16:26:25.868657 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.868661 | controller | - ' logger.go:42: 16:22:35 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.868666 | controller | -v ''^$''' 2026-03-20 16:26:25.868670 | controller | - ' logger.go:42: 16:22:35 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-20 16:26:25.868674 | controller | -1' 2026-03-20 16:26:25.868678 | controller | - ' logger.go:42: 16:22:35 | watcher/4-deploy-with-precreated-account | + APIPOD=' 2026-03-20 16:26:25.868682 | controller | - ' logger.go:42: 16:22:36 | watcher/4-deploy-with-precreated-account | running 2026-03-20 16:26:25.868687 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.868691 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.868712 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.868716 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.868720 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.868725 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.868729 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-20 16:26:25.868733 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-20 16:26:25.868737 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-20 16:26:25.868741 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.868746 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-20 16:26:25.868750 | controller | == 1 ]' 2026-03-20 16:26:25.868754 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.868758 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-20 16:26:25.868762 | controller | - ' else' 2026-03-20 16:26:25.868767 | controller | - ' exit 1' 2026-03-20 16:26:25.868771 | controller | - ' fi' 2026-03-20 16:26:25.868775 | controller | - ' ]' 2026-03-20 16:26:25.868781 | controller | - ' logger.go:42: 16:22:36 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.868786 | controller | watcher-kuttl-default' 2026-03-20 16:26:25.868790 | controller | - ' logger.go:42: 16:22:36 | watcher/4-deploy-with-precreated-account | Already 2026-03-20 16:26:25.868794 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.868798 | controller | - ' logger.go:42: 16:22:36 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-20 16:26:25.868803 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.868807 | controller | - ' logger.go:42: 16:22:36 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-20 16:26:25.868811 | controller | -1' 2026-03-20 16:26:25.868817 | controller | - ' logger.go:42: 16:22:36 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.868821 | controller | -v ''^$''' 2026-03-20 16:26:25.868826 | controller | - ' logger.go:42: 16:22:36 | watcher/4-deploy-with-precreated-account | + APIPOD=' 2026-03-20 16:26:25.868830 | controller | - ' logger.go:42: 16:22:37 | watcher/4-deploy-with-precreated-account | running 2026-03-20 16:26:25.868834 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.868838 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.868842 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.868847 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.868851 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.868855 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.868859 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-20 16:26:25.868871 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-20 16:26:25.868875 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-20 16:26:25.868879 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.868884 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-20 16:26:25.868888 | controller | == 1 ]' 2026-03-20 16:26:25.868892 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.868896 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-20 16:26:25.868901 | controller | - ' else' 2026-03-20 16:26:25.868905 | controller | - ' exit 1' 2026-03-20 16:26:25.868909 | controller | - ' fi' 2026-03-20 16:26:25.868913 | controller | - ' ]' 2026-03-20 16:26:25.868917 | controller | - ' logger.go:42: 16:22:37 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.868922 | controller | watcher-kuttl-default' 2026-03-20 16:26:25.868926 | controller | - ' logger.go:42: 16:22:38 | watcher/4-deploy-with-precreated-account | Already 2026-03-20 16:26:25.868930 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.868934 | controller | - ' logger.go:42: 16:22:38 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-20 16:26:25.868939 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.868943 | controller | - ' logger.go:42: 16:22:38 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-20 16:26:25.868947 | controller | -1' 2026-03-20 16:26:25.868951 | controller | - ' logger.go:42: 16:22:38 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.868956 | controller | -v ''^$''' 2026-03-20 16:26:25.868962 | controller | - ' logger.go:42: 16:22:38 | watcher/4-deploy-with-precreated-account | + APIPOD=' 2026-03-20 16:26:25.868966 | controller | - ' logger.go:42: 16:22:39 | watcher/4-deploy-with-precreated-account | running 2026-03-20 16:26:25.868972 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.868977 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.868981 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.868985 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.868989 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.868993 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.868998 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-20 16:26:25.869002 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-20 16:26:25.869006 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-20 16:26:25.869010 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.869015 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-20 16:26:25.869019 | controller | == 1 ]' 2026-03-20 16:26:25.869023 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.869028 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-20 16:26:25.869032 | controller | - ' else' 2026-03-20 16:26:25.869036 | controller | - ' exit 1' 2026-03-20 16:26:25.869040 | controller | - ' fi' 2026-03-20 16:26:25.869045 | controller | - ' ]' 2026-03-20 16:26:25.869049 | controller | - ' logger.go:42: 16:22:39 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.869053 | controller | watcher-kuttl-default' 2026-03-20 16:26:25.869057 | controller | - ' logger.go:42: 16:22:39 | watcher/4-deploy-with-precreated-account | Already 2026-03-20 16:26:25.869062 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.869066 | controller | - ' logger.go:42: 16:22:39 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-20 16:26:25.869070 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.869074 | controller | - ' logger.go:42: 16:22:39 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-20 16:26:25.869078 | controller | -1' 2026-03-20 16:26:25.869083 | controller | - ' logger.go:42: 16:22:39 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.869087 | controller | -v ''^$''' 2026-03-20 16:26:25.869091 | controller | - ' logger.go:42: 16:22:39 | watcher/4-deploy-with-precreated-account | + APIPOD=' 2026-03-20 16:26:25.869095 | controller | - ' logger.go:42: 16:22:40 | watcher/4-deploy-with-precreated-account | running 2026-03-20 16:26:25.869099 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.869104 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.869108 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.869112 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.869116 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.869121 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.869125 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-20 16:26:25.869138 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-20 16:26:25.869142 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-20 16:26:25.869147 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.869151 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-20 16:26:25.869157 | controller | == 1 ]' 2026-03-20 16:26:25.869161 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.869166 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-20 16:26:25.869170 | controller | - ' else' 2026-03-20 16:26:25.869174 | controller | - ' exit 1' 2026-03-20 16:26:25.869178 | controller | - ' fi' 2026-03-20 16:26:25.869182 | controller | - ' ]' 2026-03-20 16:26:25.869187 | controller | - ' logger.go:42: 16:22:40 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.869191 | controller | watcher-kuttl-default' 2026-03-20 16:26:25.869195 | controller | - ' logger.go:42: 16:22:40 | watcher/4-deploy-with-precreated-account | Already 2026-03-20 16:26:25.869199 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.869204 | controller | - ' logger.go:42: 16:22:40 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-20 16:26:25.869208 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.869212 | controller | - ' logger.go:42: 16:22:40 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-20 16:26:25.869216 | controller | -1' 2026-03-20 16:26:25.869220 | controller | - ' logger.go:42: 16:22:40 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.869225 | controller | -v ''^$''' 2026-03-20 16:26:25.869237 | controller | - ' logger.go:42: 16:22:41 | watcher/4-deploy-with-precreated-account | + APIPOD=' 2026-03-20 16:26:25.869241 | controller | - ' logger.go:42: 16:22:42 | watcher/4-deploy-with-precreated-account | running 2026-03-20 16:26:25.869245 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.869250 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.869254 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.869258 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.869262 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.869267 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.869271 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-20 16:26:25.869275 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-20 16:26:25.869279 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-20 16:26:25.869285 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.869290 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-20 16:26:25.869296 | controller | == 1 ]' 2026-03-20 16:26:25.869300 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.869305 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-20 16:26:25.869309 | controller | - ' else' 2026-03-20 16:26:25.869313 | controller | - ' exit 1' 2026-03-20 16:26:25.869318 | controller | - ' fi' 2026-03-20 16:26:25.869322 | controller | - ' ]' 2026-03-20 16:26:25.869326 | controller | - ' logger.go:42: 16:22:42 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.869331 | controller | watcher-kuttl-default' 2026-03-20 16:26:25.869335 | controller | - ' logger.go:42: 16:22:42 | watcher/4-deploy-with-precreated-account | Already 2026-03-20 16:26:25.869339 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.869344 | controller | - ' logger.go:42: 16:22:42 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-20 16:26:25.869348 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.869352 | controller | - ' logger.go:42: 16:22:42 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-20 16:26:25.869358 | controller | -1' 2026-03-20 16:26:25.869362 | controller | - ' logger.go:42: 16:22:42 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.869366 | controller | -v ''^$''' 2026-03-20 16:26:25.869371 | controller | - ' logger.go:42: 16:22:42 | watcher/4-deploy-with-precreated-account | + APIPOD=' 2026-03-20 16:26:25.869375 | controller | - ' logger.go:42: 16:22:43 | watcher/4-deploy-with-precreated-account | running 2026-03-20 16:26:25.869379 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.869383 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.869388 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.869392 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.869396 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.869400 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.869405 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-20 16:26:25.869412 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-20 16:26:25.869417 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-20 16:26:25.869428 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.869432 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-20 16:26:25.869437 | controller | == 1 ]' 2026-03-20 16:26:25.869441 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.869445 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-20 16:26:25.869449 | controller | - ' else' 2026-03-20 16:26:25.869453 | controller | - ' exit 1' 2026-03-20 16:26:25.869458 | controller | - ' fi' 2026-03-20 16:26:25.869462 | controller | - ' ]' 2026-03-20 16:26:25.869466 | controller | - ' logger.go:42: 16:22:43 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.869471 | controller | watcher-kuttl-default' 2026-03-20 16:26:25.869475 | controller | - ' logger.go:42: 16:22:43 | watcher/4-deploy-with-precreated-account | Already 2026-03-20 16:26:25.869479 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.869483 | controller | - ' logger.go:42: 16:22:43 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-20 16:26:25.869488 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.869492 | controller | - ' logger.go:42: 16:22:43 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-20 16:26:25.869496 | controller | -1' 2026-03-20 16:26:25.869500 | controller | - ' logger.go:42: 16:22:43 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.869505 | controller | -v ''^$''' 2026-03-20 16:26:25.869509 | controller | - ' logger.go:42: 16:22:43 | watcher/4-deploy-with-precreated-account | + APIPOD=' 2026-03-20 16:26:25.869513 | controller | - ' logger.go:42: 16:22:44 | watcher/4-deploy-with-precreated-account | running 2026-03-20 16:26:25.869517 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.869522 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.869526 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.869530 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.869534 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.869539 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.869543 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-20 16:26:25.869547 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-20 16:26:25.869553 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-20 16:26:25.869557 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.869562 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-20 16:26:25.869566 | controller | == 1 ]' 2026-03-20 16:26:25.869570 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.869574 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-20 16:26:25.869578 | controller | - ' else' 2026-03-20 16:26:25.869583 | controller | - ' exit 1' 2026-03-20 16:26:25.869587 | controller | - ' fi' 2026-03-20 16:26:25.869591 | controller | - ' ]' 2026-03-20 16:26:25.869595 | controller | - ' logger.go:42: 16:22:44 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.869600 | controller | watcher-kuttl-default' 2026-03-20 16:26:25.869604 | controller | - ' logger.go:42: 16:22:45 | watcher/4-deploy-with-precreated-account | Already 2026-03-20 16:26:25.869608 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.869612 | controller | - ' logger.go:42: 16:22:45 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-20 16:26:25.869617 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.869621 | controller | - ' logger.go:42: 16:22:45 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-20 16:26:25.869625 | controller | -1' 2026-03-20 16:26:25.869629 | controller | - ' logger.go:42: 16:22:45 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.869634 | controller | -v ''^$''' 2026-03-20 16:26:25.869638 | controller | - ' logger.go:42: 16:22:45 | watcher/4-deploy-with-precreated-account | + APIPOD=' 2026-03-20 16:26:25.869644 | controller | - ' logger.go:42: 16:22:46 | watcher/4-deploy-with-precreated-account | running 2026-03-20 16:26:25.869648 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.869652 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.869657 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.869661 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.869665 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.869669 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.869674 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-20 16:26:25.869678 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-20 16:26:25.869682 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-20 16:26:25.869704 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.869711 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-20 16:26:25.869715 | controller | == 1 ]' 2026-03-20 16:26:25.869720 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.869724 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-20 16:26:25.869728 | controller | - ' else' 2026-03-20 16:26:25.869733 | controller | - ' exit 1' 2026-03-20 16:26:25.869737 | controller | - ' fi' 2026-03-20 16:26:25.869741 | controller | - ' ]' 2026-03-20 16:26:25.869745 | controller | - ' logger.go:42: 16:22:46 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.869750 | controller | watcher-kuttl-default' 2026-03-20 16:26:25.869754 | controller | - ' logger.go:42: 16:22:46 | watcher/4-deploy-with-precreated-account | Already 2026-03-20 16:26:25.869760 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.869764 | controller | - ' logger.go:42: 16:22:46 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-20 16:26:25.869768 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.869773 | controller | - ' logger.go:42: 16:22:46 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-20 16:26:25.869777 | controller | -1' 2026-03-20 16:26:25.869781 | controller | - ' logger.go:42: 16:22:46 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.869785 | controller | -v ''^$''' 2026-03-20 16:26:25.869789 | controller | - ' logger.go:42: 16:22:46 | watcher/4-deploy-with-precreated-account | + APIPOD=' 2026-03-20 16:26:25.869794 | controller | - ' logger.go:42: 16:22:47 | watcher/4-deploy-with-precreated-account | running 2026-03-20 16:26:25.869798 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.869802 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.869806 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.869811 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.869815 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.869819 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.869823 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-20 16:26:25.869827 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-20 16:26:25.869832 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-20 16:26:25.869836 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.869840 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-20 16:26:25.869844 | controller | == 1 ]' 2026-03-20 16:26:25.869849 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.869853 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-20 16:26:25.869857 | controller | - ' else' 2026-03-20 16:26:25.869861 | controller | - ' exit 1' 2026-03-20 16:26:25.869865 | controller | - ' fi' 2026-03-20 16:26:25.869870 | controller | - ' ]' 2026-03-20 16:26:25.869874 | controller | - ' logger.go:42: 16:22:47 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.869878 | controller | watcher-kuttl-default' 2026-03-20 16:26:25.869882 | controller | - ' logger.go:42: 16:22:47 | watcher/4-deploy-with-precreated-account | Already 2026-03-20 16:26:25.869887 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.869891 | controller | - ' logger.go:42: 16:22:47 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.869895 | controller | -v ''^$''' 2026-03-20 16:26:25.869899 | controller | - ' logger.go:42: 16:22:47 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-20 16:26:25.869903 | controller | -1' 2026-03-20 16:26:25.869908 | controller | - ' logger.go:42: 16:22:47 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-20 16:26:25.869912 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.869916 | controller | - ' logger.go:42: 16:22:48 | watcher/4-deploy-with-precreated-account | + APIPOD=' 2026-03-20 16:26:25.869920 | controller | - ' logger.go:42: 16:22:49 | watcher/4-deploy-with-precreated-account | running 2026-03-20 16:26:25.869924 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.869929 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.869933 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.869937 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.869943 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.869947 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.869952 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-20 16:26:25.869956 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-20 16:26:25.869960 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-20 16:26:25.869964 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.869975 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-20 16:26:25.869980 | controller | == 1 ]' 2026-03-20 16:26:25.869984 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.869988 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-20 16:26:25.869993 | controller | - ' else' 2026-03-20 16:26:25.869997 | controller | - ' exit 1' 2026-03-20 16:26:25.870001 | controller | - ' fi' 2026-03-20 16:26:25.870005 | controller | - ' ]' 2026-03-20 16:26:25.870010 | controller | - ' logger.go:42: 16:22:49 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.870014 | controller | watcher-kuttl-default' 2026-03-20 16:26:25.870018 | controller | - ' logger.go:42: 16:22:49 | watcher/4-deploy-with-precreated-account | Already 2026-03-20 16:26:25.870022 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.870027 | controller | - ' logger.go:42: 16:22:49 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-20 16:26:25.870031 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.870035 | controller | - ' logger.go:42: 16:22:49 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-20 16:26:25.870039 | controller | -1' 2026-03-20 16:26:25.870043 | controller | - ' logger.go:42: 16:22:49 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.870048 | controller | -v ''^$''' 2026-03-20 16:26:25.870052 | controller | - ' logger.go:42: 16:22:49 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' 2026-03-20 16:26:25.870056 | controller | - ' logger.go:42: 16:22:49 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.870060 | controller | -n watcher-kuttl-api-0 '']''' 2026-03-20 16:26:25.870064 | controller | - ' logger.go:42: 16:22:49 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.870069 | controller | -c ''^# Global config''' 2026-03-20 16:26:25.870075 | controller | - ' logger.go:42: 16:22:49 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-20 16:26:25.870079 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' 2026-03-20 16:26:25.870083 | controller | - ' logger.go:42: 16:22:49 | watcher/4-deploy-with-precreated-account | error: 2026-03-20 16:26:25.870087 | controller | unable to upgrade connection: container not found ("watcher-api")' 2026-03-20 16:26:25.870092 | controller | - ' logger.go:42: 16:22:49 | watcher/4-deploy-with-precreated-account | ++ echo' 2026-03-20 16:26:25.870096 | controller | - ' logger.go:42: 16:22:49 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.870100 | controller | 0 == 1 '']''' 2026-03-20 16:26:25.870104 | controller | - ' logger.go:42: 16:22:50 | watcher/4-deploy-with-precreated-account | running 2026-03-20 16:26:25.870108 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.870113 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.870117 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.870121 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.870125 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.870131 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.870135 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-20 16:26:25.870139 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-20 16:26:25.870144 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-20 16:26:25.870148 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.870152 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-20 16:26:25.870156 | controller | == 1 ]' 2026-03-20 16:26:25.870160 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.870165 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-20 16:26:25.870169 | controller | - ' else' 2026-03-20 16:26:25.870173 | controller | - ' exit 1' 2026-03-20 16:26:25.870177 | controller | - ' fi' 2026-03-20 16:26:25.870181 | controller | - ' ]' 2026-03-20 16:26:25.870186 | controller | - ' logger.go:42: 16:22:50 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.870192 | controller | watcher-kuttl-default' 2026-03-20 16:26:25.870196 | controller | - ' logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account | Already 2026-03-20 16:26:25.870200 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.870204 | controller | - ' logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-20 16:26:25.870209 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.870213 | controller | - ' logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-20 16:26:25.870217 | controller | -1' 2026-03-20 16:26:25.870221 | controller | - ' logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.870225 | controller | -v ''^$''' 2026-03-20 16:26:25.870230 | controller | - ' logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' 2026-03-20 16:26:25.870240 | controller | - ' logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.870245 | controller | -n watcher-kuttl-api-0 '']''' 2026-03-20 16:26:25.870249 | controller | - ' logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.870253 | controller | -c ''^# Global config''' 2026-03-20 16:26:25.870258 | controller | - ' logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-20 16:26:25.870262 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' 2026-03-20 16:26:25.870266 | controller | - ' logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.870270 | controller | ''#'' Global config' 2026-03-20 16:26:25.870275 | controller | - ' logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.870279 | controller | 1 == 1 '']''' 2026-03-20 16:26:25.870283 | controller | - ' logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.870287 | controller | -c ''^# Service config''' 2026-03-20 16:26:25.870291 | controller | - ' logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-20 16:26:25.870296 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' 2026-03-20 16:26:25.870300 | controller | - ' logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.870304 | controller | ''#'' Service config' 2026-03-20 16:26:25.870308 | controller | - ' logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.870312 | controller | 1 == 1 '']''' 2026-03-20 16:26:25.870318 | controller | - ' logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.870323 | controller | -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' 2026-03-20 16:26:25.870327 | controller | - ' logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-20 16:26:25.870331 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.870335 | controller | - ' logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.870340 | controller | ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.870344 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.870348 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.870352 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_test:4341713c0331ec810a387bb85d10e1d1@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-20 16:26:25.870357 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-20 16:26:25.870361 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.870365 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.870369 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.870374 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.870378 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.870382 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.870386 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.870390 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.870395 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-20 16:26:25.870399 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.870405 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.870409 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-20 16:26:25.870413 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-20 16:26:25.870418 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.870422 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.870428 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.870432 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-20 16:26:25.870436 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.870446 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' 2026-03-20 16:26:25.870451 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-20 16:26:25.870455 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-20 16:26:25.870461 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-20 16:26:25.870465 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-20 16:26:25.870469 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-20 16:26:25.870474 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-20 16:26:25.870478 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-20 16:26:25.870482 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-20 16:26:25.870486 | controller | - ' logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.870490 | controller | 1 == 1 '']''' 2026-03-20 16:26:25.870495 | controller | - ' logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.870499 | controller | -czPo ''TimeOut 80''' 2026-03-20 16:26:25.870505 | controller | - ' logger.go:42: 16:22:51 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-20 16:26:25.870509 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' 2026-03-20 16:26:25.870513 | controller | - ' logger.go:42: 16:22:52 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.870518 | controller | ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-20 16:26:25.870526 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-20 16:26:25.870530 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-20 16:26:25.870534 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-20 16:26:25.870538 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-20 16:26:25.870543 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-20 16:26:25.870547 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-20 16:26:25.870551 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-20 16:26:25.870555 | controller | internal display-name=internal group=watcher processes=2 threads=1 user=watcher 2026-03-20 16:26:25.870559 | controller | WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' 2026-03-20 16:26:25.870564 | controller | ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-20 16:26:25.870572 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-20 16:26:25.870576 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-20 16:26:25.870580 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-20 16:26:25.870584 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-20 16:26:25.870589 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-20 16:26:25.870593 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-20 16:26:25.870597 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-20 16:26:25.870601 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-20 16:26:25.870605 | controller | public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' 2026-03-20 16:26:25.870613 | controller | - ' logger.go:42: 16:22:52 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.870617 | controller | 1 == 1 '']''' 2026-03-20 16:26:25.870621 | controller | - ' logger.go:42: 16:22:53 | watcher/4-deploy-with-precreated-account | running 2026-03-20 16:26:25.870625 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.870630 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.870634 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.870638 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.870642 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.870646 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.870651 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-20 16:26:25.870655 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-20 16:26:25.870666 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-20 16:26:25.870670 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.870675 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-20 16:26:25.870679 | controller | == 1 ]' 2026-03-20 16:26:25.870683 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.870687 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-20 16:26:25.870707 | controller | - ' else' 2026-03-20 16:26:25.870713 | controller | - ' exit 1' 2026-03-20 16:26:25.870717 | controller | - ' fi' 2026-03-20 16:26:25.870722 | controller | - ' ]' 2026-03-20 16:26:25.870726 | controller | - ' logger.go:42: 16:22:53 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.870730 | controller | watcher-kuttl-default' 2026-03-20 16:26:25.870734 | controller | - ' logger.go:42: 16:22:53 | watcher/4-deploy-with-precreated-account | Already 2026-03-20 16:26:25.870739 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.870743 | controller | - ' logger.go:42: 16:22:53 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-20 16:26:25.870747 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.870751 | controller | - ' logger.go:42: 16:22:53 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-20 16:26:25.870755 | controller | -1' 2026-03-20 16:26:25.870760 | controller | - ' logger.go:42: 16:22:53 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.870764 | controller | -v ''^$''' 2026-03-20 16:26:25.870768 | controller | - ' logger.go:42: 16:22:53 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' 2026-03-20 16:26:25.870772 | controller | - ' logger.go:42: 16:22:53 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.870776 | controller | -n watcher-kuttl-api-0 '']''' 2026-03-20 16:26:25.870780 | controller | - ' logger.go:42: 16:22:53 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.870785 | controller | -c ''^# Global config''' 2026-03-20 16:26:25.870789 | controller | - ' logger.go:42: 16:22:53 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-20 16:26:25.870793 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' 2026-03-20 16:26:25.870797 | controller | - ' logger.go:42: 16:22:53 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.870801 | controller | ''#'' Global config' 2026-03-20 16:26:25.870806 | controller | - ' logger.go:42: 16:22:53 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.870810 | controller | 1 == 1 '']''' 2026-03-20 16:26:25.870814 | controller | - ' logger.go:42: 16:22:53 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.870821 | controller | -c ''^# Service config''' 2026-03-20 16:26:25.870826 | controller | - ' logger.go:42: 16:22:53 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-20 16:26:25.870830 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' 2026-03-20 16:26:25.870834 | controller | - ' logger.go:42: 16:22:54 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.870838 | controller | ''#'' Service config' 2026-03-20 16:26:25.870842 | controller | - ' logger.go:42: 16:22:54 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.870847 | controller | 1 == 1 '']''' 2026-03-20 16:26:25.870851 | controller | - ' logger.go:42: 16:22:54 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.870855 | controller | -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' 2026-03-20 16:26:25.870859 | controller | - ' logger.go:42: 16:22:54 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-20 16:26:25.870863 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.870867 | controller | - ' logger.go:42: 16:22:54 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.870872 | controller | ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.870876 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.870880 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.870886 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_test:4341713c0331ec810a387bb85d10e1d1@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-20 16:26:25.870891 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-20 16:26:25.870895 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.870899 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.870911 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.870915 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.870919 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.870924 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.870928 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.870932 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.870936 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-20 16:26:25.870940 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.870944 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.870949 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-20 16:26:25.870954 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-20 16:26:25.870959 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.870963 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.870969 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.870973 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-20 16:26:25.870977 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.870981 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' 2026-03-20 16:26:25.870985 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-20 16:26:25.870990 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-20 16:26:25.870994 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-20 16:26:25.870998 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-20 16:26:25.871002 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-20 16:26:25.871006 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-20 16:26:25.871011 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-20 16:26:25.871015 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-20 16:26:25.871019 | controller | - ' logger.go:42: 16:22:54 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.871023 | controller | 1 == 1 '']''' 2026-03-20 16:26:25.871027 | controller | - ' logger.go:42: 16:22:54 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.871031 | controller | -czPo ''TimeOut 80''' 2026-03-20 16:26:25.871036 | controller | - ' logger.go:42: 16:22:54 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-20 16:26:25.871040 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' 2026-03-20 16:26:25.871044 | controller | - ' logger.go:42: 16:22:54 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.871048 | controller | ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-20 16:26:25.871056 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-20 16:26:25.871061 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-20 16:26:25.871065 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-20 16:26:25.871069 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-20 16:26:25.871073 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-20 16:26:25.871077 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-20 16:26:25.871081 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-20 16:26:25.871138 | controller | internal display-name=internal group=watcher processes=2 threads=1 user=watcher 2026-03-20 16:26:25.871146 | controller | WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' 2026-03-20 16:26:25.871151 | controller | ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-20 16:26:25.871159 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-20 16:26:25.871170 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-20 16:26:25.871175 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-20 16:26:25.871181 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-20 16:26:25.871185 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-20 16:26:25.871189 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-20 16:26:25.871194 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-20 16:26:25.871198 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-20 16:26:25.871202 | controller | public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' 2026-03-20 16:26:25.871206 | controller | - ' logger.go:42: 16:22:54 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.871210 | controller | 1 == 1 '']''' 2026-03-20 16:26:25.871216 | controller | - ' logger.go:42: 16:22:55 | watcher/4-deploy-with-precreated-account | running 2026-03-20 16:26:25.871220 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.871225 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.871229 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.871233 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.871237 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.871241 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.871246 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-20 16:26:25.871250 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-20 16:26:25.871254 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-20 16:26:25.871258 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.871262 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-20 16:26:25.871267 | controller | == 1 ]' 2026-03-20 16:26:25.871271 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.871275 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-20 16:26:25.871279 | controller | - ' else' 2026-03-20 16:26:25.871283 | controller | - ' exit 1' 2026-03-20 16:26:25.871288 | controller | - ' fi' 2026-03-20 16:26:25.871292 | controller | - ' ]' 2026-03-20 16:26:25.871296 | controller | - ' logger.go:42: 16:22:55 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.871300 | controller | watcher-kuttl-default' 2026-03-20 16:26:25.871305 | controller | - ' logger.go:42: 16:22:55 | watcher/4-deploy-with-precreated-account | Already 2026-03-20 16:26:25.871309 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.871313 | controller | - ' logger.go:42: 16:22:55 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-20 16:26:25.871317 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.871321 | controller | - ' logger.go:42: 16:22:55 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-20 16:26:25.871325 | controller | -1' 2026-03-20 16:26:25.871330 | controller | - ' logger.go:42: 16:22:55 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.871334 | controller | -v ''^$''' 2026-03-20 16:26:25.871338 | controller | - ' logger.go:42: 16:22:56 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' 2026-03-20 16:26:25.871342 | controller | - ' logger.go:42: 16:22:56 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.871346 | controller | -n watcher-kuttl-api-0 '']''' 2026-03-20 16:26:25.871350 | controller | - ' logger.go:42: 16:22:56 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.871356 | controller | -c ''^# Global config''' 2026-03-20 16:26:25.871361 | controller | - ' logger.go:42: 16:22:56 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-20 16:26:25.871365 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' 2026-03-20 16:26:25.871369 | controller | - ' logger.go:42: 16:22:56 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.871373 | controller | ''#'' Global config' 2026-03-20 16:26:25.871377 | controller | - ' logger.go:42: 16:22:56 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.871382 | controller | 1 == 1 '']''' 2026-03-20 16:26:25.871386 | controller | - ' logger.go:42: 16:22:56 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.871390 | controller | -c ''^# Service config''' 2026-03-20 16:26:25.871394 | controller | - ' logger.go:42: 16:22:56 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-20 16:26:25.871398 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' 2026-03-20 16:26:25.871403 | controller | - ' logger.go:42: 16:22:56 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.871407 | controller | ''#'' Service config' 2026-03-20 16:26:25.871411 | controller | - ' logger.go:42: 16:22:56 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.871415 | controller | 1 == 1 '']''' 2026-03-20 16:26:25.871425 | controller | - ' logger.go:42: 16:22:56 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.871430 | controller | -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' 2026-03-20 16:26:25.871434 | controller | - ' logger.go:42: 16:22:56 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-20 16:26:25.871438 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.871443 | controller | - ' logger.go:42: 16:22:56 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.871447 | controller | ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.871451 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.871455 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.871459 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_test:4341713c0331ec810a387bb85d10e1d1@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-20 16:26:25.871464 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-20 16:26:25.871468 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.871472 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.871476 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.871480 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.871485 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.871489 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.871493 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.871497 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.871501 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-20 16:26:25.871507 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.871511 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.871516 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-20 16:26:25.871520 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-20 16:26:25.871524 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.871528 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.871533 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.871537 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-20 16:26:25.871543 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.871547 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' 2026-03-20 16:26:25.871551 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-20 16:26:25.871555 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-20 16:26:25.871559 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-20 16:26:25.871564 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-20 16:26:25.871568 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-20 16:26:25.871572 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-20 16:26:25.871576 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-20 16:26:25.871580 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-20 16:26:25.871584 | controller | - ' logger.go:42: 16:22:56 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.871589 | controller | 1 == 1 '']''' 2026-03-20 16:26:25.871593 | controller | - ' logger.go:42: 16:22:56 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.871597 | controller | -czPo ''TimeOut 80''' 2026-03-20 16:26:25.871601 | controller | - ' logger.go:42: 16:22:56 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-20 16:26:25.871612 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' 2026-03-20 16:26:25.871617 | controller | - ' logger.go:42: 16:22:56 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.871621 | controller | ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-20 16:26:25.871629 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-20 16:26:25.871633 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-20 16:26:25.871638 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-20 16:26:25.871642 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-20 16:26:25.871646 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-20 16:26:25.871650 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-20 16:26:25.871654 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-20 16:26:25.871658 | controller | internal display-name=internal group=watcher processes=2 threads=1 user=watcher 2026-03-20 16:26:25.871664 | controller | WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' 2026-03-20 16:26:25.871668 | controller | ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-20 16:26:25.871677 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-20 16:26:25.871681 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-20 16:26:25.871685 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-20 16:26:25.871689 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-20 16:26:25.871707 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-20 16:26:25.871711 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-20 16:26:25.871716 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-20 16:26:25.871720 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-20 16:26:25.871724 | controller | public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' 2026-03-20 16:26:25.871728 | controller | - ' logger.go:42: 16:22:56 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.871732 | controller | 1 == 1 '']''' 2026-03-20 16:26:25.871736 | controller | - ' logger.go:42: 16:22:57 | watcher/4-deploy-with-precreated-account | running 2026-03-20 16:26:25.871741 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.871745 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.871749 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.871753 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.871759 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.871764 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.871768 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-20 16:26:25.871773 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-20 16:26:25.871777 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-20 16:26:25.871781 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.871804 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-20 16:26:25.871809 | controller | == 1 ]' 2026-03-20 16:26:25.871813 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.871817 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-20 16:26:25.871822 | controller | - ' else' 2026-03-20 16:26:25.871826 | controller | - ' exit 1' 2026-03-20 16:26:25.871830 | controller | - ' fi' 2026-03-20 16:26:25.871834 | controller | - ' ]' 2026-03-20 16:26:25.871840 | controller | - ' logger.go:42: 16:22:57 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.871845 | controller | watcher-kuttl-default' 2026-03-20 16:26:25.871849 | controller | - ' logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account | Already 2026-03-20 16:26:25.871853 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.871857 | controller | - ' logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-20 16:26:25.871861 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.871867 | controller | - ' logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-20 16:26:25.871871 | controller | -1' 2026-03-20 16:26:25.871884 | controller | - ' logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.871888 | controller | -v ''^$''' 2026-03-20 16:26:25.871896 | controller | - ' logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' 2026-03-20 16:26:25.871900 | controller | - ' logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.871904 | controller | -n watcher-kuttl-api-0 '']''' 2026-03-20 16:26:25.871908 | controller | - ' logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.871913 | controller | -c ''^# Global config''' 2026-03-20 16:26:25.871917 | controller | - ' logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-20 16:26:25.871921 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' 2026-03-20 16:26:25.871925 | controller | - ' logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.871929 | controller | ''#'' Global config' 2026-03-20 16:26:25.871934 | controller | - ' logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.871938 | controller | 1 == 1 '']''' 2026-03-20 16:26:25.871942 | controller | - ' logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.871946 | controller | -c ''^# Service config''' 2026-03-20 16:26:25.871950 | controller | - ' logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-20 16:26:25.871954 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' 2026-03-20 16:26:25.871959 | controller | - ' logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.871963 | controller | ''#'' Service config' 2026-03-20 16:26:25.871967 | controller | - ' logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.871971 | controller | 1 == 1 '']''' 2026-03-20 16:26:25.871975 | controller | - ' logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.871980 | controller | -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' 2026-03-20 16:26:25.871984 | controller | - ' logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-20 16:26:25.871988 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.871992 | controller | - ' logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.871997 | controller | ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.872001 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.872005 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.872009 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_test:4341713c0331ec810a387bb85d10e1d1@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-20 16:26:25.872013 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-20 16:26:25.872018 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.872022 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.872026 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.872032 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.872036 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.872040 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.872044 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.872049 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.872053 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-20 16:26:25.872057 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.872061 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.872065 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-20 16:26:25.872070 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-20 16:26:25.872074 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.872078 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.872089 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.872093 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-20 16:26:25.872098 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.872102 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' 2026-03-20 16:26:25.872108 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-20 16:26:25.872112 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-20 16:26:25.872116 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-20 16:26:25.872121 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-20 16:26:25.872125 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-20 16:26:25.872129 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-20 16:26:25.872133 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-20 16:26:25.872138 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-20 16:26:25.872142 | controller | - ' logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.872146 | controller | 1 == 1 '']''' 2026-03-20 16:26:25.872150 | controller | - ' logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.872154 | controller | -czPo ''TimeOut 80''' 2026-03-20 16:26:25.872159 | controller | - ' logger.go:42: 16:22:58 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-20 16:26:25.872163 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' 2026-03-20 16:26:25.872167 | controller | - ' logger.go:42: 16:22:59 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.872171 | controller | ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-20 16:26:25.872179 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-20 16:26:25.872184 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-20 16:26:25.872189 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-20 16:26:25.872194 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-20 16:26:25.872198 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-20 16:26:25.872202 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-20 16:26:25.872206 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-20 16:26:25.872210 | controller | internal display-name=internal group=watcher processes=2 threads=1 user=watcher 2026-03-20 16:26:25.872214 | controller | WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' 2026-03-20 16:26:25.872219 | controller | ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-20 16:26:25.872227 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-20 16:26:25.872231 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-20 16:26:25.872235 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-20 16:26:25.872239 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-20 16:26:25.872244 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-20 16:26:25.872248 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-20 16:26:25.872252 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-20 16:26:25.872256 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-20 16:26:25.872260 | controller | public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' 2026-03-20 16:26:25.872265 | controller | - ' logger.go:42: 16:22:59 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.872269 | controller | 1 == 1 '']''' 2026-03-20 16:26:25.872273 | controller | - ' logger.go:42: 16:23:00 | watcher/4-deploy-with-precreated-account | running 2026-03-20 16:26:25.872277 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.872282 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.872286 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.872290 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.872294 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.872305 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.872309 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-20 16:26:25.872314 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-20 16:26:25.872318 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-20 16:26:25.872322 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.872326 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-20 16:26:25.872331 | controller | == 1 ]' 2026-03-20 16:26:25.872335 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.872339 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-20 16:26:25.872343 | controller | - ' else' 2026-03-20 16:26:25.872347 | controller | - ' exit 1' 2026-03-20 16:26:25.872353 | controller | - ' fi' 2026-03-20 16:26:25.872357 | controller | - ' ]' 2026-03-20 16:26:25.872362 | controller | - ' logger.go:42: 16:23:00 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.872366 | controller | watcher-kuttl-default' 2026-03-20 16:26:25.872370 | controller | - ' logger.go:42: 16:23:00 | watcher/4-deploy-with-precreated-account | Already 2026-03-20 16:26:25.872376 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.872380 | controller | - ' logger.go:42: 16:23:00 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-20 16:26:25.872384 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.872388 | controller | - ' logger.go:42: 16:23:00 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.872393 | controller | -v ''^$''' 2026-03-20 16:26:25.872397 | controller | - ' logger.go:42: 16:23:00 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-20 16:26:25.872401 | controller | -1' 2026-03-20 16:26:25.872405 | controller | - ' logger.go:42: 16:23:00 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' 2026-03-20 16:26:25.872409 | controller | - ' logger.go:42: 16:23:00 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.872413 | controller | -n watcher-kuttl-api-0 '']''' 2026-03-20 16:26:25.872418 | controller | - ' logger.go:42: 16:23:00 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.872422 | controller | -c ''^# Global config''' 2026-03-20 16:26:25.872426 | controller | - ' logger.go:42: 16:23:00 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-20 16:26:25.872430 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' 2026-03-20 16:26:25.872434 | controller | - ' logger.go:42: 16:23:00 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.872439 | controller | ''#'' Global config' 2026-03-20 16:26:25.872443 | controller | - ' logger.go:42: 16:23:00 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.872447 | controller | 1 == 1 '']''' 2026-03-20 16:26:25.872451 | controller | - ' logger.go:42: 16:23:00 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.872455 | controller | -c ''^# Service config''' 2026-03-20 16:26:25.872460 | controller | - ' logger.go:42: 16:23:00 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-20 16:26:25.872464 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' 2026-03-20 16:26:25.872468 | controller | - ' logger.go:42: 16:23:00 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.872472 | controller | ''#'' Service config' 2026-03-20 16:26:25.872476 | controller | - ' logger.go:42: 16:23:00 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.872480 | controller | 1 == 1 '']''' 2026-03-20 16:26:25.872485 | controller | - ' logger.go:42: 16:23:00 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.872489 | controller | -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' 2026-03-20 16:26:25.872493 | controller | - ' logger.go:42: 16:23:00 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-20 16:26:25.872497 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.872501 | controller | - ' logger.go:42: 16:23:01 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.872506 | controller | ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.872510 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.872514 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.872518 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_test:4341713c0331ec810a387bb85d10e1d1@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-20 16:26:25.872526 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-20 16:26:25.872537 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.872541 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.872545 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.872550 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.872554 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.872558 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.872562 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.872566 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.872571 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-20 16:26:25.872575 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.872579 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.872583 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-20 16:26:25.872587 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-20 16:26:25.872592 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.872596 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.872600 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.872604 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-20 16:26:25.872608 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.872613 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' 2026-03-20 16:26:25.872617 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-20 16:26:25.872621 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-20 16:26:25.872627 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-20 16:26:25.872631 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-20 16:26:25.872635 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-20 16:26:25.872639 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-20 16:26:25.872644 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-20 16:26:25.872648 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-20 16:26:25.872652 | controller | - ' logger.go:42: 16:23:01 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.872656 | controller | 1 == 1 '']''' 2026-03-20 16:26:25.872660 | controller | - ' logger.go:42: 16:23:01 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.872665 | controller | -czPo ''TimeOut 80''' 2026-03-20 16:26:25.872669 | controller | - ' logger.go:42: 16:23:01 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-20 16:26:25.872674 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' 2026-03-20 16:26:25.872679 | controller | - ' logger.go:42: 16:23:01 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.872683 | controller | ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-20 16:26:25.872702 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-20 16:26:25.872708 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-20 16:26:25.872712 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-20 16:26:25.872717 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-20 16:26:25.872721 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-20 16:26:25.872725 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-20 16:26:25.872729 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-20 16:26:25.872733 | controller | internal display-name=internal group=watcher processes=2 threads=1 user=watcher 2026-03-20 16:26:25.872738 | controller | WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' 2026-03-20 16:26:25.872749 | controller | ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-20 16:26:25.872758 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-20 16:26:25.872762 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-20 16:26:25.872766 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-20 16:26:25.872771 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-20 16:26:25.872775 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-20 16:26:25.872779 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-20 16:26:25.872783 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-20 16:26:25.872787 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-20 16:26:25.872791 | controller | public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' 2026-03-20 16:26:25.872796 | controller | - ' logger.go:42: 16:23:01 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.872800 | controller | 1 == 1 '']''' 2026-03-20 16:26:25.872804 | controller | - ' logger.go:42: 16:23:02 | watcher/4-deploy-with-precreated-account | running 2026-03-20 16:26:25.872808 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.872812 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.872817 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.872821 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.872825 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.872829 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.872834 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-20 16:26:25.872838 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-20 16:26:25.872842 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-20 16:26:25.872856 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.872861 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-20 16:26:25.872865 | controller | == 1 ]' 2026-03-20 16:26:25.872869 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.872873 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-20 16:26:25.872877 | controller | - ' else' 2026-03-20 16:26:25.872882 | controller | - ' exit 1' 2026-03-20 16:26:25.872886 | controller | - ' fi' 2026-03-20 16:26:25.872890 | controller | - ' ]' 2026-03-20 16:26:25.872894 | controller | - ' logger.go:42: 16:23:02 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.872899 | controller | watcher-kuttl-default' 2026-03-20 16:26:25.872903 | controller | - ' logger.go:42: 16:23:02 | watcher/4-deploy-with-precreated-account | Already 2026-03-20 16:26:25.872907 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.872912 | controller | - ' logger.go:42: 16:23:02 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.872916 | controller | -v ''^$''' 2026-03-20 16:26:25.872922 | controller | - ' logger.go:42: 16:23:02 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-20 16:26:25.872928 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.872932 | controller | - ' logger.go:42: 16:23:02 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-20 16:26:25.872936 | controller | -1' 2026-03-20 16:26:25.872940 | controller | - ' logger.go:42: 16:23:02 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' 2026-03-20 16:26:25.872945 | controller | - ' logger.go:42: 16:23:02 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.872949 | controller | -n watcher-kuttl-api-0 '']''' 2026-03-20 16:26:25.872953 | controller | - ' logger.go:42: 16:23:02 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.872957 | controller | -c ''^# Global config''' 2026-03-20 16:26:25.872962 | controller | - ' logger.go:42: 16:23:02 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-20 16:26:25.872966 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' 2026-03-20 16:26:25.872970 | controller | - ' logger.go:42: 16:23:02 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.872974 | controller | ''#'' Global config' 2026-03-20 16:26:25.872979 | controller | - ' logger.go:42: 16:23:02 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.872983 | controller | 1 == 1 '']''' 2026-03-20 16:26:25.872987 | controller | - ' logger.go:42: 16:23:02 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.872991 | controller | -c ''^# Service config''' 2026-03-20 16:26:25.872996 | controller | - ' logger.go:42: 16:23:02 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-20 16:26:25.873000 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' 2026-03-20 16:26:25.873010 | controller | - ' logger.go:42: 16:23:03 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.873015 | controller | ''#'' Service config' 2026-03-20 16:26:25.873019 | controller | - ' logger.go:42: 16:23:03 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.873023 | controller | 1 == 1 '']''' 2026-03-20 16:26:25.873028 | controller | - ' logger.go:42: 16:23:03 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.873034 | controller | -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' 2026-03-20 16:26:25.873038 | controller | - ' logger.go:42: 16:23:03 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-20 16:26:25.873044 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.873048 | controller | - ' logger.go:42: 16:23:03 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.873052 | controller | ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.873057 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.873061 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.873065 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_test:4341713c0331ec810a387bb85d10e1d1@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-20 16:26:25.873069 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-20 16:26:25.873073 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.873077 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.873082 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.873086 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.873090 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.873094 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.873098 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.873103 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.873107 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-20 16:26:25.873111 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.873115 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.873119 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-20 16:26:25.873123 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-20 16:26:25.873128 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.873132 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.873136 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.873140 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-20 16:26:25.873144 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.873149 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' 2026-03-20 16:26:25.873153 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-20 16:26:25.873157 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-20 16:26:25.873161 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-20 16:26:25.873168 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-20 16:26:25.873173 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-20 16:26:25.873177 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-20 16:26:25.873183 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-20 16:26:25.873188 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-20 16:26:25.873199 | controller | - ' logger.go:42: 16:23:03 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.873203 | controller | 1 == 1 '']''' 2026-03-20 16:26:25.873207 | controller | - ' logger.go:42: 16:23:03 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.873212 | controller | -czPo ''TimeOut 80''' 2026-03-20 16:26:25.873216 | controller | - ' logger.go:42: 16:23:03 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-20 16:26:25.873220 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' 2026-03-20 16:26:25.873224 | controller | - ' logger.go:42: 16:23:03 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.873228 | controller | ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-20 16:26:25.873237 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-20 16:26:25.873241 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-20 16:26:25.873245 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-20 16:26:25.873249 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-20 16:26:25.873254 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-20 16:26:25.873258 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-20 16:26:25.873262 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-20 16:26:25.873266 | controller | internal display-name=internal group=watcher processes=2 threads=1 user=watcher 2026-03-20 16:26:25.873270 | controller | WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' 2026-03-20 16:26:25.873274 | controller | ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-20 16:26:25.873283 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-20 16:26:25.873287 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-20 16:26:25.873291 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-20 16:26:25.873295 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-20 16:26:25.873299 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-20 16:26:25.873303 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-20 16:26:25.873308 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-20 16:26:25.873312 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-20 16:26:25.873316 | controller | public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' 2026-03-20 16:26:25.873320 | controller | - ' logger.go:42: 16:23:03 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.873324 | controller | 1 == 1 '']''' 2026-03-20 16:26:25.873329 | controller | - ' logger.go:42: 16:23:04 | watcher/4-deploy-with-precreated-account | running 2026-03-20 16:26:25.873333 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.873337 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.873343 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.873347 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.873351 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.873355 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.873360 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-20 16:26:25.873364 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-20 16:26:25.873368 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-20 16:26:25.873372 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.873376 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-20 16:26:25.873381 | controller | == 1 ]' 2026-03-20 16:26:25.873385 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.873389 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-20 16:26:25.873393 | controller | - ' else' 2026-03-20 16:26:25.873397 | controller | - ' exit 1' 2026-03-20 16:26:25.873402 | controller | - ' fi' 2026-03-20 16:26:25.873406 | controller | - ' ]' 2026-03-20 16:26:25.873410 | controller | - ' logger.go:42: 16:23:04 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.873414 | controller | watcher-kuttl-default' 2026-03-20 16:26:25.873418 | controller | - ' logger.go:42: 16:23:04 | watcher/4-deploy-with-precreated-account | Already 2026-03-20 16:26:25.873423 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.873433 | controller | - ' logger.go:42: 16:23:04 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-20 16:26:25.873437 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.873442 | controller | - ' logger.go:42: 16:23:04 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-20 16:26:25.873446 | controller | -1' 2026-03-20 16:26:25.873462 | controller | - ' logger.go:42: 16:23:04 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.873467 | controller | -v ''^$''' 2026-03-20 16:26:25.873471 | controller | - ' logger.go:42: 16:23:04 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' 2026-03-20 16:26:25.873475 | controller | - ' logger.go:42: 16:23:04 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.873479 | controller | -n watcher-kuttl-api-0 '']''' 2026-03-20 16:26:25.873483 | controller | - ' logger.go:42: 16:23:04 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.873488 | controller | -c ''^# Global config''' 2026-03-20 16:26:25.873492 | controller | - ' logger.go:42: 16:23:04 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-20 16:26:25.873496 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' 2026-03-20 16:26:25.873500 | controller | - ' logger.go:42: 16:23:05 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.873504 | controller | ''#'' Global config' 2026-03-20 16:26:25.873509 | controller | - ' logger.go:42: 16:23:05 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.873513 | controller | 1 == 1 '']''' 2026-03-20 16:26:25.873517 | controller | - ' logger.go:42: 16:23:05 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.873521 | controller | -c ''^# Service config''' 2026-03-20 16:26:25.873525 | controller | - ' logger.go:42: 16:23:05 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-20 16:26:25.873530 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' 2026-03-20 16:26:25.873534 | controller | - ' logger.go:42: 16:23:05 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.873540 | controller | ''#'' Service config' 2026-03-20 16:26:25.873544 | controller | - ' logger.go:42: 16:23:05 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.873548 | controller | 1 == 1 '']''' 2026-03-20 16:26:25.873552 | controller | - ' logger.go:42: 16:23:05 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-20 16:26:25.873556 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.873560 | controller | - ' logger.go:42: 16:23:05 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.873565 | controller | -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' 2026-03-20 16:26:25.873569 | controller | - ' logger.go:42: 16:23:05 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.873573 | controller | ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.873577 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.873582 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.873586 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_test:4341713c0331ec810a387bb85d10e1d1@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-20 16:26:25.873590 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-20 16:26:25.873594 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.873598 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.873603 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.873607 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.873611 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.873615 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.873619 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.873623 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.873628 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-20 16:26:25.873632 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.873636 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.873640 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-20 16:26:25.873651 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-20 16:26:25.873655 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.873660 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.873664 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.873668 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-20 16:26:25.873673 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.873677 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' 2026-03-20 16:26:25.873683 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-20 16:26:25.873687 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-20 16:26:25.873702 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-20 16:26:25.873708 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-20 16:26:25.873712 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-20 16:26:25.873716 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-20 16:26:25.873721 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-20 16:26:25.873725 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-20 16:26:25.873729 | controller | - ' logger.go:42: 16:23:05 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.873733 | controller | 1 == 1 '']''' 2026-03-20 16:26:25.873740 | controller | - ' logger.go:42: 16:23:05 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.873744 | controller | -czPo ''TimeOut 80''' 2026-03-20 16:26:25.873748 | controller | - ' logger.go:42: 16:23:05 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-20 16:26:25.873752 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' 2026-03-20 16:26:25.873756 | controller | - ' logger.go:42: 16:23:05 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.873760 | controller | ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-20 16:26:25.873769 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-20 16:26:25.873773 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-20 16:26:25.873777 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-20 16:26:25.873781 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-20 16:26:25.873785 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-20 16:26:25.873789 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-20 16:26:25.873794 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-20 16:26:25.873798 | controller | internal display-name=internal group=watcher processes=2 threads=1 user=watcher 2026-03-20 16:26:25.873802 | controller | WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' 2026-03-20 16:26:25.873806 | controller | ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-20 16:26:25.873815 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-20 16:26:25.873819 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-20 16:26:25.873823 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-20 16:26:25.873827 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-20 16:26:25.873831 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-20 16:26:25.873836 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-20 16:26:25.873841 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-20 16:26:25.873847 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-20 16:26:25.873851 | controller | public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' 2026-03-20 16:26:25.873855 | controller | - ' logger.go:42: 16:23:05 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.873860 | controller | 1 == 1 '']''' 2026-03-20 16:26:25.873864 | controller | - ' logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account | running 2026-03-20 16:26:25.873876 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.873881 | controller | - ' oc project watcher-kuttl-default' 2026-03-20 16:26:25.873885 | controller | - ' APIPOD=$(oc get pods -n watcher-kuttl-default -l "service=watcher-api" 2026-03-20 16:26:25.873889 | controller | -ocustom-columns=:metadata.name|grep -v ^$|head -1)' 2026-03-20 16:26:25.873894 | controller | - ' if [ -n "${APIPOD}" ]; then' 2026-03-20 16:26:25.873898 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/01-global-custom.conf) 2026-03-20 16:26:25.873902 | controller | |grep -c "^# Global config") == 1 ]' 2026-03-20 16:26:25.873906 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/02-service-custom.conf) 2026-03-20 16:26:25.873911 | controller | |grep -c "^# Service config") == 1 ]' 2026-03-20 16:26:25.873915 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/watcher/watcher.conf.d/00-default.conf) 2026-03-20 16:26:25.873919 | controller | |grep -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem'') 2026-03-20 16:26:25.873923 | controller | == 1 ]' 2026-03-20 16:26:25.873928 | controller | - ' [ $(echo $(oc rsh -c watcher-api ${APIPOD} cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf) 2026-03-20 16:26:25.873932 | controller | |grep -czPo ''TimeOut 80'') == 1 ]' 2026-03-20 16:26:25.873936 | controller | - ' else' 2026-03-20 16:26:25.873940 | controller | - ' exit 1' 2026-03-20 16:26:25.873945 | controller | - ' fi' 2026-03-20 16:26:25.873949 | controller | - ' ]' 2026-03-20 16:26:25.873953 | controller | - ' logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account | + oc project 2026-03-20 16:26:25.873957 | controller | watcher-kuttl-default' 2026-03-20 16:26:25.873961 | controller | - ' logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account | Already 2026-03-20 16:26:25.873966 | controller | on project "watcher-kuttl-default" on server "https://api.crc.testing:6443".' 2026-03-20 16:26:25.873970 | controller | - ' logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account | ++ oc get 2026-03-20 16:26:25.873974 | controller | pods -n watcher-kuttl-default -l service=watcher-api -ocustom-columns=:metadata.name' 2026-03-20 16:26:25.873978 | controller | - ' logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account | ++ head 2026-03-20 16:26:25.873982 | controller | -1' 2026-03-20 16:26:25.873986 | controller | - ' logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.873991 | controller | -v ''^$''' 2026-03-20 16:26:25.873995 | controller | - ' logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account | + APIPOD=watcher-kuttl-api-0' 2026-03-20 16:26:25.873999 | controller | - ' logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.874003 | controller | -n watcher-kuttl-api-0 '']''' 2026-03-20 16:26:25.874009 | controller | - ' logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.874013 | controller | -c ''^# Global config''' 2026-03-20 16:26:25.874018 | controller | - ' logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-20 16:26:25.874022 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/01-global-custom.conf' 2026-03-20 16:26:25.874026 | controller | - ' logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.874032 | controller | ''#'' Global config' 2026-03-20 16:26:25.874036 | controller | - ' logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.874040 | controller | 1 == 1 '']''' 2026-03-20 16:26:25.874044 | controller | - ' logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.874049 | controller | -c ''^# Service config''' 2026-03-20 16:26:25.874053 | controller | - ' logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-20 16:26:25.874057 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/02-service-custom.conf' 2026-03-20 16:26:25.874061 | controller | - ' logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.874065 | controller | ''#'' Service config' 2026-03-20 16:26:25.874070 | controller | - ' logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.874074 | controller | 1 == 1 '']''' 2026-03-20 16:26:25.874078 | controller | - ' logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.874082 | controller | -czPo ''\[prometheus_client\]\s+host\s+=\s+metric-storage-prometheus.watcher-kuttl-default.svc\s+port\s+=\s+9090\s+cafile\s+=\s+/etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem''' 2026-03-20 16:26:25.874086 | controller | - ' logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-20 16:26:25.874090 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/watcher/watcher.conf.d/00-default.conf' 2026-03-20 16:26:25.874095 | controller | - ' logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.874099 | controller | ''[DEFAULT]'' state_path = /var/lib/watcher transport_url = ''rabbit://default_user_yf15aOSfqh1lP-rMLgj:9ik4H84CQBhJph2TWkj8zaS96s0e3C6e@rabbitmq.watcher-kuttl-default.svc:5671/?ssl=1'' 2026-03-20 16:26:25.874103 | controller | control_exchange = watcher debug = True log_file = /var/log/watcher/watcher-kuttl-api.log 2026-03-20 16:26:25.874107 | controller | ''#'' empty notification_level means that no notification will be sent notification_level 2026-03-20 16:26:25.874118 | controller | = ''[database]'' connection = ''mysql+pymysql://watcher_test:4341713c0331ec810a387bb85d10e1d1@openstack.watcher-kuttl-default.svc/watcher?read_default_file=/etc/my.cnf'' 2026-03-20 16:26:25.874123 | controller | ''[oslo_policy]'' policy_file = /etc/watcher/policy.yaml.sample ''[oslo_messaging_notifications]'' 2026-03-20 16:26:25.874127 | controller | driver = noop ''[oslo_messaging_rabbit]'' rabbit_quorum_queue=true rabbit_transient_quorum_queue=true 2026-03-20 16:26:25.874133 | controller | amqp_durable_queues=true ''[keystone_authtoken]'' memcached_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.874137 | controller | tls_enabled=true memcache_tls_certfile = /etc/pki/tls/certs/mtls.crt memcache_tls_keyfile 2026-03-20 16:26:25.874141 | controller | = /etc/pki/tls/private/mtls.key memcache_tls_cafile = /etc/pki/tls/certs/mtls-ca.crt 2026-03-20 16:26:25.874146 | controller | memcache_tls_enabled = true project_domain_name = Default project_name = service 2026-03-20 16:26:25.874150 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.874154 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.874158 | controller | internal region_name = regionOne cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem 2026-03-20 16:26:25.874162 | controller | ''[watcher_clients_auth]'' project_domain_name = Default project_name = service 2026-03-20 16:26:25.874166 | controller | user_domain_name = Default password = password username = watcher auth_type = password 2026-03-20 16:26:25.874171 | controller | auth_url = https://keystone-internal.watcher-kuttl-default.svc:5000 interface = 2026-03-20 16:26:25.874175 | controller | internal cafile = /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ''[oslo_concurrency]'' 2026-03-20 16:26:25.874179 | controller | lock_path = /var/lib/watcher/tmp ''[watcher_datasources]'' datasources = prometheus 2026-03-20 16:26:25.874185 | controller | ''[cache]'' backend = oslo_cache.memcache_pool memcache_servers=memcached-0.memcached.watcher-kuttl-default.svc:11212 2026-03-20 16:26:25.874189 | controller | memcache_socket_timeout = 0.5 memcache_pool_connection_get_timeout = 1 enabled=true 2026-03-20 16:26:25.874193 | controller | tls_enabled=true tls_certfile=/etc/pki/tls/certs/mtls.crt tls_keyfile=/etc/pki/tls/private/mtls.key 2026-03-20 16:26:25.874197 | controller | tls_cafile=/etc/pki/tls/certs/mtls-ca.crt memcache_dead_retry = 30 ''[prometheus_client]'' 2026-03-20 16:26:25.874201 | controller | host = metric-storage-prometheus.watcher-kuttl-default.svc port = 9090 cafile = 2026-03-20 16:26:25.874206 | controller | /etc/pki/ca-trust/extracted/pem/prometheus/internal-ca-bundle.pem ''[aetos_client]'' 2026-03-20 16:26:25.874210 | controller | endpoint_type = internal region_name = regionOne ''[cinder_client]'' endpoint_type 2026-03-20 16:26:25.874214 | controller | = internal region_name = regionOne ''[glance_client]'' endpoint_type = internal 2026-03-20 16:26:25.874218 | controller | region_name = regionOne ''[ironic_client]'' endpoint_type = internal region_name 2026-03-20 16:26:25.874222 | controller | = regionOne ''[keystone_client]'' interface = internal region_name = regionOne ''[neutron_client]'' 2026-03-20 16:26:25.874226 | controller | endpoint_type = internal region_name = regionOne ''[nova_client]'' endpoint_type 2026-03-20 16:26:25.874231 | controller | = internal region_name = regionOne ''[placement_client]'' interface = internal region_name 2026-03-20 16:26:25.874235 | controller | = regionOne ''[watcher_cluster_data_model_collectors.compute]'' period = 900 ''[watcher_cluster_data_model_collectors.baremetal]'' 2026-03-20 16:26:25.874239 | controller | period = 900 ''[watcher_cluster_data_model_collectors.storage]'' period = 900' 2026-03-20 16:26:25.874243 | controller | - ' logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.874247 | controller | 1 == 1 '']''' 2026-03-20 16:26:25.874252 | controller | - ' logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account | ++ grep 2026-03-20 16:26:25.874256 | controller | -czPo ''TimeOut 80''' 2026-03-20 16:26:25.874260 | controller | - ' logger.go:42: 16:23:07 | watcher/4-deploy-with-precreated-account | +++ oc 2026-03-20 16:26:25.874266 | controller | rsh -c watcher-api watcher-kuttl-api-0 cat /etc/httpd/conf.d/10-watcher-wsgi-main.conf' 2026-03-20 16:26:25.874270 | controller | - ' logger.go:42: 16:23:08 | watcher/4-deploy-with-precreated-account | ++ echo 2026-03-20 16:26:25.874274 | controller | ''#'' internal vhost watcher-internal.watcher-kuttl-default.svc configuration '''' ServerName watcher-internal.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-20 16:26:25.874285 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-20 16:26:25.874290 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-20 16:26:25.874296 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-20 16:26:25.874311 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-20 16:26:25.874319 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-20 16:26:25.874325 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-20 16:26:25.874343 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-20 16:26:25.874350 | controller | internal display-name=internal group=watcher processes=2 threads=1 user=watcher 2026-03-20 16:26:25.874356 | controller | WSGIProcessGroup internal WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' '''' 2026-03-20 16:26:25.874362 | controller | ''#'' public vhost watcher-public.watcher-kuttl-default.svc configuration '''' ServerName watcher-public.watcher-kuttl-default.svc ''##'' Vhost docroot 2026-03-20 16:26:25.874374 | controller | DocumentRoot ''"/var/www/cgi-bin"'' ''#'' Set the timeout for the watcher-api TimeOut 2026-03-20 16:26:25.874386 | controller | 80 ''##'' Directories, there should at least be a declaration for /var/www/cgi-bin 2026-03-20 16:26:25.874391 | controller | '''' Options -Indexes +FollowSymLinks +MultiViews 2026-03-20 16:26:25.874397 | controller | AllowOverride None Require all granted '''' ''##'' Logging ErrorLog 2026-03-20 16:26:25.874403 | controller | /dev/stdout ServerSignature Off CustomLog /dev/stdout combined ''env=!forwarded'' 2026-03-20 16:26:25.874408 | controller | CustomLog /dev/stdout proxy env=forwarded ''##'' set watcher log level to debug 2026-03-20 16:26:25.874414 | controller | LogLevel debug ''##'' WSGI configuration WSGIApplicationGroup ''%{GLOBAL}'' WSGIDaemonProcess 2026-03-20 16:26:25.874420 | controller | public display-name=public group=watcher processes=2 threads=1 user=watcher WSGIProcessGroup 2026-03-20 16:26:25.874425 | controller | public WSGIScriptAlias / ''"/usr/bin/watcher-api-wsgi"'' ''''' 2026-03-20 16:26:25.874431 | controller | - ' logger.go:42: 16:23:08 | watcher/4-deploy-with-precreated-account | + ''['' 2026-03-20 16:26:25.874437 | controller | 1 == 1 '']''' 2026-03-20 16:26:25.874443 | controller | - ' logger.go:42: 16:23:08 | watcher/4-deploy-with-precreated-account | test step 2026-03-20 16:26:25.874448 | controller | completed 4-deploy-with-precreated-account' 2026-03-20 16:26:25.874454 | controller | - ' logger.go:42: 16:23:08 | watcher/5-cleanup-watcher | starting test step 5-cleanup-watcher' 2026-03-20 16:26:25.874460 | controller | - ' logger.go:42: 16:23:08 | watcher/5-cleanup-watcher | running command: [sh -c 2026-03-20 16:26:25.874465 | controller | set -ex' 2026-03-20 16:26:25.874471 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.874477 | controller | -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]' 2026-03-20 16:26:25.874482 | controller | - ' ]' 2026-03-20 16:26:25.874488 | controller | - ' logger.go:42: 16:23:08 | watcher/5-cleanup-watcher | + oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.874493 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-20 16:26:25.874499 | controller | - ' logger.go:42: 16:23:08 | watcher/5-cleanup-watcher | ++ grep -c ''^watcher''' 2026-03-20 16:26:25.874504 | controller | - ' logger.go:42: 16:23:11 | watcher/5-cleanup-watcher | + ''['' 1 == 0 '']''' 2026-03-20 16:26:25.874510 | controller | - ' logger.go:42: 16:23:12 | watcher/5-cleanup-watcher | running command: [sh -c 2026-03-20 16:26:25.874515 | controller | set -ex' 2026-03-20 16:26:25.874520 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.874526 | controller | -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]' 2026-03-20 16:26:25.874532 | controller | - ' ]' 2026-03-20 16:26:25.874537 | controller | - ' logger.go:42: 16:23:12 | watcher/5-cleanup-watcher | + oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.874543 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-20 16:26:25.874549 | controller | - ' logger.go:42: 16:23:12 | watcher/5-cleanup-watcher | ++ grep -c ''^watcher''' 2026-03-20 16:26:25.874555 | controller | - ' logger.go:42: 16:23:14 | watcher/5-cleanup-watcher | + ''['' 0 == 0 '']''' 2026-03-20 16:26:25.874560 | controller | - ' logger.go:42: 16:23:15 | watcher/5-cleanup-watcher | running command: [sh -c 2026-03-20 16:26:25.874566 | controller | set -ex' 2026-03-20 16:26:25.874572 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.874582 | controller | -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]' 2026-03-20 16:26:25.874588 | controller | - ' ]' 2026-03-20 16:26:25.874594 | controller | - ' logger.go:42: 16:23:15 | watcher/5-cleanup-watcher | + oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.874599 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-20 16:26:25.874606 | controller | - ' logger.go:42: 16:23:15 | watcher/5-cleanup-watcher | ++ grep -c ''^watcher''' 2026-03-20 16:26:25.874612 | controller | - ' logger.go:42: 16:23:17 | watcher/5-cleanup-watcher | + ''['' 0 == 0 '']''' 2026-03-20 16:26:25.874620 | controller | - ' logger.go:42: 16:23:18 | watcher/5-cleanup-watcher | running command: [sh -c 2026-03-20 16:26:25.874625 | controller | set -ex' 2026-03-20 16:26:25.874629 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.874634 | controller | -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]' 2026-03-20 16:26:25.874638 | controller | - ' ]' 2026-03-20 16:26:25.874644 | controller | - ' logger.go:42: 16:23:18 | watcher/5-cleanup-watcher | + oc exec -n watcher-kuttl-default 2026-03-20 16:26:25.874648 | controller | openstackclient -- openstack service list -f value -c Name -c Type' 2026-03-20 16:26:25.874653 | controller | - ' logger.go:42: 16:23:18 | watcher/5-cleanup-watcher | ++ grep -c ''^watcher''' 2026-03-20 16:26:25.874657 | controller | - ' logger.go:42: 16:23:21 | watcher/5-cleanup-watcher | + ''['' 0 == 0 '']''' 2026-03-20 16:26:25.874673 | controller | - ' logger.go:42: 16:23:21 | watcher/5-cleanup-watcher | test step completed 5-cleanup-watcher' 2026-03-20 16:26:25.874678 | controller | - ' logger.go:42: 16:23:21 | watcher | skipping kubernetes event logging' 2026-03-20 16:26:25.874683 | controller | - === CONT kuttl/harness/watcher-cinder 2026-03-20 16:26:25.874687 | controller | - ' logger.go:42: 16:23:21 | watcher-cinder | Skipping creation of user-supplied 2026-03-20 16:26:25.874705 | controller | namespace: watcher-kuttl-default' 2026-03-20 16:26:25.874713 | controller | - ' logger.go:42: 16:23:21 | watcher-cinder/0-cleanup-watcher | starting test step 2026-03-20 16:26:25.874717 | controller | 0-cleanup-watcher' 2026-03-20 16:26:25.874722 | controller | - ' logger.go:42: 16:23:21 | watcher-cinder/0-cleanup-watcher | test step completed 2026-03-20 16:26:25.874726 | controller | 0-cleanup-watcher' 2026-03-20 16:26:25.874730 | controller | - ' logger.go:42: 16:23:21 | watcher-cinder/1-deploy-watcher-no-cinder | starting 2026-03-20 16:26:25.874734 | controller | test step 1-deploy-watcher-no-cinder' 2026-03-20 16:26:25.874738 | controller | - ' logger.go:42: 16:23:21 | watcher-cinder/1-deploy-watcher-no-cinder | Watcher:watcher-kuttl-default/watcher-kuttl 2026-03-20 16:26:25.874743 | controller | created' 2026-03-20 16:26:25.874747 | controller | - ' logger.go:42: 16:23:21 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-20 16:26:25.874751 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.874755 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.874759 | controller | service' 2026-03-20 16:26:25.874763 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-20 16:26:25.874768 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.874772 | controller | - ' ]' 2026-03-20 16:26:25.874776 | controller | - ' logger.go:42: 16:23:21 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-20 16:26:25.874780 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.874784 | controller | - ' logger.go:42: 16:23:21 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-20 16:26:25.874788 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.874793 | controller | - ' logger.go:42: 16:23:21 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-20 16:26:25.874797 | controller | error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in 2026-03-20 16:26:25.874801 | controller | namespace "watcher-kuttl-default"' 2026-03-20 16:26:25.874805 | controller | - ' logger.go:42: 16:23:21 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-20 16:26:25.874809 | controller | 0 == 2 '']''' 2026-03-20 16:26:25.874813 | controller | - ' logger.go:42: 16:23:22 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-20 16:26:25.874818 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.874822 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.874826 | controller | service' 2026-03-20 16:26:25.874832 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-20 16:26:25.874836 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.874841 | controller | - ' ]' 2026-03-20 16:26:25.874845 | controller | - ' logger.go:42: 16:23:22 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-20 16:26:25.874849 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.874853 | controller | - ' logger.go:42: 16:23:22 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-20 16:26:25.874857 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.874861 | controller | - ' logger.go:42: 16:23:22 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-20 16:26:25.874865 | controller | error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in 2026-03-20 16:26:25.874870 | controller | namespace "watcher-kuttl-default"' 2026-03-20 16:26:25.874874 | controller | - ' logger.go:42: 16:23:22 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-20 16:26:25.874878 | controller | 0 == 2 '']''' 2026-03-20 16:26:25.874882 | controller | - ' logger.go:42: 16:23:23 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-20 16:26:25.874886 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.874891 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.874895 | controller | service' 2026-03-20 16:26:25.874899 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-20 16:26:25.874903 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.874907 | controller | - ' ]' 2026-03-20 16:26:25.874911 | controller | - ' logger.go:42: 16:23:23 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-20 16:26:25.874916 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.874920 | controller | - ' logger.go:42: 16:23:23 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-20 16:26:25.874924 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.874938 | controller | - ' logger.go:42: 16:23:23 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-20 16:26:25.874943 | controller | error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in 2026-03-20 16:26:25.874947 | controller | namespace "watcher-kuttl-default"' 2026-03-20 16:26:25.874952 | controller | - ' logger.go:42: 16:23:23 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-20 16:26:25.874956 | controller | 0 == 2 '']''' 2026-03-20 16:26:25.874968 | controller | - ' logger.go:42: 16:23:24 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-20 16:26:25.874973 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.874978 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.874983 | controller | service' 2026-03-20 16:26:25.874989 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-20 16:26:25.874995 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.875000 | controller | - ' ]' 2026-03-20 16:26:25.875006 | controller | - ' logger.go:42: 16:23:24 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-20 16:26:25.875012 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.875017 | controller | - ' logger.go:42: 16:23:24 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-20 16:26:25.875023 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.875028 | controller | - ' logger.go:42: 16:23:24 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-20 16:26:25.875033 | controller | error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in 2026-03-20 16:26:25.875039 | controller | namespace "watcher-kuttl-default"' 2026-03-20 16:26:25.875045 | controller | - ' logger.go:42: 16:23:24 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-20 16:26:25.875054 | controller | 0 == 2 '']''' 2026-03-20 16:26:25.875060 | controller | - ' logger.go:42: 16:23:25 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-20 16:26:25.875066 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.875072 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.875077 | controller | service' 2026-03-20 16:26:25.875083 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-20 16:26:25.875089 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.875096 | controller | - ' ]' 2026-03-20 16:26:25.875102 | controller | - ' logger.go:42: 16:23:25 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-20 16:26:25.875108 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.875113 | controller | - ' logger.go:42: 16:23:25 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-20 16:26:25.875119 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.875125 | controller | - ' logger.go:42: 16:23:25 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-20 16:26:25.875131 | controller | error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in 2026-03-20 16:26:25.875137 | controller | namespace "watcher-kuttl-default"' 2026-03-20 16:26:25.875143 | controller | - ' logger.go:42: 16:23:25 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-20 16:26:25.875148 | controller | 0 == 2 '']''' 2026-03-20 16:26:25.875154 | controller | - ' logger.go:42: 16:23:26 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-20 16:26:25.875160 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.875166 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.875171 | controller | service' 2026-03-20 16:26:25.875177 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-20 16:26:25.875182 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.875188 | controller | - ' ]' 2026-03-20 16:26:25.875193 | controller | - ' logger.go:42: 16:23:26 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-20 16:26:25.875198 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.875204 | controller | - ' logger.go:42: 16:23:26 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-20 16:26:25.875209 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.875214 | controller | - ' logger.go:42: 16:23:27 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-20 16:26:25.875218 | controller | error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in 2026-03-20 16:26:25.875224 | controller | namespace "watcher-kuttl-default"' 2026-03-20 16:26:25.875230 | controller | - ' logger.go:42: 16:23:27 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-20 16:26:25.875236 | controller | 0 == 2 '']''' 2026-03-20 16:26:25.875242 | controller | - ' logger.go:42: 16:23:28 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-20 16:26:25.875247 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.875254 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.875259 | controller | service' 2026-03-20 16:26:25.875265 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-20 16:26:25.875271 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.875277 | controller | - ' ]' 2026-03-20 16:26:25.875282 | controller | - ' logger.go:42: 16:23:28 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-20 16:26:25.875288 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.875297 | controller | - ' logger.go:42: 16:23:28 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-20 16:26:25.875306 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.875311 | controller | - ' logger.go:42: 16:23:28 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-20 16:26:25.875317 | controller | error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in 2026-03-20 16:26:25.875335 | controller | namespace "watcher-kuttl-default"' 2026-03-20 16:26:25.875347 | controller | - ' logger.go:42: 16:23:28 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-20 16:26:25.875353 | controller | 0 == 2 '']''' 2026-03-20 16:26:25.875359 | controller | - ' logger.go:42: 16:23:29 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-20 16:26:25.875365 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.875370 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.875376 | controller | service' 2026-03-20 16:26:25.875382 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-20 16:26:25.875390 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.875396 | controller | - ' ]' 2026-03-20 16:26:25.875402 | controller | - ' logger.go:42: 16:23:29 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-20 16:26:25.875407 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.875412 | controller | - ' logger.go:42: 16:23:29 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-20 16:26:25.875418 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.875424 | controller | - ' logger.go:42: 16:23:29 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-20 16:26:25.875430 | controller | error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in 2026-03-20 16:26:25.875436 | controller | namespace "watcher-kuttl-default"' 2026-03-20 16:26:25.875442 | controller | - ' logger.go:42: 16:23:29 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-20 16:26:25.875448 | controller | 0 == 2 '']''' 2026-03-20 16:26:25.875453 | controller | - ' logger.go:42: 16:23:30 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-20 16:26:25.875459 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.875465 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.875471 | controller | service' 2026-03-20 16:26:25.875477 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-20 16:26:25.875482 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.875488 | controller | - ' ]' 2026-03-20 16:26:25.875494 | controller | - ' logger.go:42: 16:23:30 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-20 16:26:25.875499 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.875505 | controller | - ' logger.go:42: 16:23:30 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-20 16:26:25.875512 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.875518 | controller | - ' logger.go:42: 16:23:30 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-20 16:26:25.875523 | controller | error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in 2026-03-20 16:26:25.875529 | controller | namespace "watcher-kuttl-default"' 2026-03-20 16:26:25.875535 | controller | - ' logger.go:42: 16:23:30 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-20 16:26:25.875541 | controller | 0 == 2 '']''' 2026-03-20 16:26:25.875547 | controller | - ' logger.go:42: 16:23:31 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-20 16:26:25.875553 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.875558 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.875564 | controller | service' 2026-03-20 16:26:25.875570 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-20 16:26:25.875579 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.875585 | controller | - ' ]' 2026-03-20 16:26:25.875591 | controller | - ' logger.go:42: 16:23:31 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-20 16:26:25.875597 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.875602 | controller | - ' logger.go:42: 16:23:31 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-20 16:26:25.875607 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.875613 | controller | - ' logger.go:42: 16:23:31 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-20 16:26:25.875618 | controller | error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in 2026-03-20 16:26:25.875624 | controller | namespace "watcher-kuttl-default"' 2026-03-20 16:26:25.875629 | controller | - ' logger.go:42: 16:23:31 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-20 16:26:25.875635 | controller | 0 == 2 '']''' 2026-03-20 16:26:25.875641 | controller | - ' logger.go:42: 16:23:32 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-20 16:26:25.875646 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.875652 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.875658 | controller | service' 2026-03-20 16:26:25.875664 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-20 16:26:25.875670 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.875675 | controller | - ' ]' 2026-03-20 16:26:25.875684 | controller | - ' logger.go:42: 16:23:32 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-20 16:26:25.875713 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.875725 | controller | - ' logger.go:42: 16:23:32 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-20 16:26:25.875731 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.875748 | controller | - ' logger.go:42: 16:23:32 | watcher-cinder/1-deploy-watcher-no-cinder | error: 2026-03-20 16:26:25.875754 | controller | error from server (NotFound): pods "watcher-kuttl-decision-engine-0" not found in 2026-03-20 16:26:25.875760 | controller | namespace "watcher-kuttl-default"' 2026-03-20 16:26:25.875765 | controller | - ' logger.go:42: 16:23:32 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-20 16:26:25.875771 | controller | 0 == 2 '']''' 2026-03-20 16:26:25.875776 | controller | - ' logger.go:42: 16:23:33 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-20 16:26:25.875782 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.875787 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.875815 | controller | service' 2026-03-20 16:26:25.875822 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-20 16:26:25.875828 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.875833 | controller | - ' ]' 2026-03-20 16:26:25.875839 | controller | - ' logger.go:42: 16:23:33 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-20 16:26:25.875845 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.875851 | controller | - ' logger.go:42: 16:23:33 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-20 16:26:25.875856 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.875862 | controller | - ' logger.go:42: 16:23:33 | watcher-cinder/1-deploy-watcher-no-cinder | Error 2026-03-20 16:26:25.875868 | controller | from server (BadRequest): container "watcher-decision-engine" in pod "watcher-kuttl-decision-engine-0" 2026-03-20 16:26:25.875874 | controller | is waiting to start: ContainerCreating' 2026-03-20 16:26:25.875879 | controller | - ' logger.go:42: 16:23:33 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-20 16:26:25.875893 | controller | 0 == 2 '']''' 2026-03-20 16:26:25.875899 | controller | - ' logger.go:42: 16:23:34 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-20 16:26:25.875905 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.875911 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.875917 | controller | service' 2026-03-20 16:26:25.875922 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-20 16:26:25.875928 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.875934 | controller | - ' ]' 2026-03-20 16:26:25.875940 | controller | - ' logger.go:42: 16:23:34 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-20 16:26:25.875945 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.875951 | controller | - ' logger.go:42: 16:23:34 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-20 16:26:25.875957 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.875963 | controller | - ' logger.go:42: 16:23:35 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-20 16:26:25.875968 | controller | 0 == 2 '']''' 2026-03-20 16:26:25.875974 | controller | - ' logger.go:42: 16:23:36 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-20 16:26:25.875979 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.875985 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.875990 | controller | service' 2026-03-20 16:26:25.875996 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-20 16:26:25.876002 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.876007 | controller | - ' ]' 2026-03-20 16:26:25.876013 | controller | - ' logger.go:42: 16:23:36 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-20 16:26:25.876019 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.876025 | controller | - ' logger.go:42: 16:23:36 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-20 16:26:25.876031 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.876036 | controller | - ' logger.go:42: 16:23:36 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-20 16:26:25.876042 | controller | 0 == 2 '']''' 2026-03-20 16:26:25.876048 | controller | - ' logger.go:42: 16:23:37 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-20 16:26:25.876054 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.876060 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.876065 | controller | service' 2026-03-20 16:26:25.876071 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-20 16:26:25.876076 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.876082 | controller | - ' ]' 2026-03-20 16:26:25.876088 | controller | - ' logger.go:42: 16:23:37 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-20 16:26:25.876093 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.876099 | controller | - ' logger.go:42: 16:23:37 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-20 16:26:25.876105 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.876110 | controller | - ' logger.go:42: 16:23:37 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-20 16:26:25.876116 | controller | 1 == 2 '']''' 2026-03-20 16:26:25.876121 | controller | - ' logger.go:42: 16:23:38 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-20 16:26:25.876127 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.876148 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.876157 | controller | service' 2026-03-20 16:26:25.876163 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-20 16:26:25.876169 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.876175 | controller | - ' ]' 2026-03-20 16:26:25.876181 | controller | - ' logger.go:42: 16:23:38 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-20 16:26:25.876186 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.876192 | controller | - ' logger.go:42: 16:23:38 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-20 16:26:25.876198 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.876204 | controller | - ' logger.go:42: 16:23:38 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-20 16:26:25.876210 | controller | 2 == 2 '']''' 2026-03-20 16:26:25.876216 | controller | - ' logger.go:42: 16:23:39 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-20 16:26:25.876222 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.876228 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.876234 | controller | service' 2026-03-20 16:26:25.876240 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-20 16:26:25.876246 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.876251 | controller | - ' ]' 2026-03-20 16:26:25.876257 | controller | - ' logger.go:42: 16:23:39 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-20 16:26:25.876263 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.876269 | controller | - ' logger.go:42: 16:23:39 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-20 16:26:25.876275 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.876281 | controller | - ' logger.go:42: 16:23:39 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-20 16:26:25.876287 | controller | 2 == 2 '']''' 2026-03-20 16:26:25.876293 | controller | - ' logger.go:42: 16:23:40 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-20 16:26:25.876299 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.876305 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.876311 | controller | service' 2026-03-20 16:26:25.876316 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-20 16:26:25.876322 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.876328 | controller | - ' ]' 2026-03-20 16:26:25.876334 | controller | - ' logger.go:42: 16:23:40 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-20 16:26:25.876340 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.876346 | controller | - ' logger.go:42: 16:23:40 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-20 16:26:25.876352 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.876358 | controller | - ' logger.go:42: 16:23:41 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-20 16:26:25.876364 | controller | 2 == 2 '']''' 2026-03-20 16:26:25.876370 | controller | - ' logger.go:42: 16:23:42 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-20 16:26:25.876375 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.876381 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.876387 | controller | service' 2026-03-20 16:26:25.876393 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-20 16:26:25.876399 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.876405 | controller | - ' ]' 2026-03-20 16:26:25.876411 | controller | - ' logger.go:42: 16:23:42 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-20 16:26:25.876420 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.876426 | controller | - ' logger.go:42: 16:23:42 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-20 16:26:25.876432 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.876438 | controller | - ' logger.go:42: 16:23:42 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-20 16:26:25.876443 | controller | 2 == 2 '']''' 2026-03-20 16:26:25.876449 | controller | - ' logger.go:42: 16:23:43 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-20 16:26:25.876456 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.876465 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.876471 | controller | service' 2026-03-20 16:26:25.876477 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-20 16:26:25.876483 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.876488 | controller | - ' ]' 2026-03-20 16:26:25.876494 | controller | - ' logger.go:42: 16:23:43 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-20 16:26:25.876500 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.876506 | controller | - ' logger.go:42: 16:23:43 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-20 16:26:25.876512 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.876528 | controller | - ' logger.go:42: 16:23:43 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-20 16:26:25.876535 | controller | 2 == 2 '']''' 2026-03-20 16:26:25.876541 | controller | - ' logger.go:42: 16:23:44 | watcher-cinder/1-deploy-watcher-no-cinder | running 2026-03-20 16:26:25.876546 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.876552 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.876558 | controller | service' 2026-03-20 16:26:25.876566 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 | grep -c ''Block 2026-03-20 16:26:25.876572 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.876578 | controller | - ' ]' 2026-03-20 16:26:25.876584 | controller | - ' logger.go:42: 16:23:44 | watcher-cinder/1-deploy-watcher-no-cinder | ++ oc 2026-03-20 16:26:25.876590 | controller | logs -n watcher-kuttl-default watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.876596 | controller | - ' logger.go:42: 16:23:44 | watcher-cinder/1-deploy-watcher-no-cinder | ++ grep 2026-03-20 16:26:25.876602 | controller | -c ''Block storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.876608 | controller | - ' logger.go:42: 16:23:44 | watcher-cinder/1-deploy-watcher-no-cinder | + ''['' 2026-03-20 16:26:25.876614 | controller | 2 == 2 '']''' 2026-03-20 16:26:25.876619 | controller | - ' logger.go:42: 16:23:44 | watcher-cinder/1-deploy-watcher-no-cinder | test step 2026-03-20 16:26:25.876625 | controller | completed 1-deploy-watcher-no-cinder' 2026-03-20 16:26:25.876631 | controller | - ' logger.go:42: 16:23:44 | watcher-cinder/2-deploy-cinder | starting test step 2026-03-20 16:26:25.876637 | controller | 2-deploy-cinder' 2026-03-20 16:26:25.876643 | controller | - ' logger.go:42: 16:23:44 | watcher-cinder/2-deploy-cinder | OpenStackControlPlane:watcher-kuttl-default/openstack 2026-03-20 16:26:25.876649 | controller | updated' 2026-03-20 16:26:25.876655 | controller | - ' logger.go:42: 16:23:44 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.876661 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.876667 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.876673 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.876679 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.876684 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.876709 | controller | - ' ]' 2026-03-20 16:26:25.876718 | controller | - ' logger.go:42: 16:23:44 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.876724 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.876730 | controller | - ' logger.go:42: 16:23:44 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.876736 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.876742 | controller | - ' logger.go:42: 16:23:45 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.876748 | controller | - ' logger.go:42: 16:23:46 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.876754 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.876760 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.876766 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.876772 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.876778 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.876783 | controller | - ' ]' 2026-03-20 16:26:25.876789 | controller | - ' logger.go:42: 16:23:46 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.876795 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.876801 | controller | - ' logger.go:42: 16:23:46 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.876807 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.876813 | controller | - ' logger.go:42: 16:23:46 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.876819 | controller | - ' logger.go:42: 16:23:47 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.876825 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.876831 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.876837 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.876843 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.876849 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.876854 | controller | - ' ]' 2026-03-20 16:26:25.876860 | controller | - ' logger.go:42: 16:23:47 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.876866 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.876872 | controller | - ' logger.go:42: 16:23:47 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.876878 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.876884 | controller | - ' logger.go:42: 16:23:47 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.876890 | controller | - ' logger.go:42: 16:23:48 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.876896 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.876902 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.876908 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.876923 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.876930 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.876935 | controller | - ' ]' 2026-03-20 16:26:25.876941 | controller | - ' logger.go:42: 16:23:48 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.876947 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.876953 | controller | - ' logger.go:42: 16:23:48 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.876959 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.876968 | controller | - ' logger.go:42: 16:23:48 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.876974 | controller | - ' logger.go:42: 16:23:49 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.876979 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.876985 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.876991 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.876997 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.877003 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.877009 | controller | - ' ]' 2026-03-20 16:26:25.877015 | controller | - ' logger.go:42: 16:23:49 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.877021 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.877027 | controller | - ' logger.go:42: 16:23:49 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.877032 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.877038 | controller | - ' logger.go:42: 16:23:49 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.877047 | controller | - ' logger.go:42: 16:23:50 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.877053 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.877059 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.877065 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.877071 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.877076 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.877082 | controller | - ' ]' 2026-03-20 16:26:25.877087 | controller | - ' logger.go:42: 16:23:50 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.877092 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.877098 | controller | - ' logger.go:42: 16:23:50 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.877103 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.877109 | controller | - ' logger.go:42: 16:23:50 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.877114 | controller | - ' logger.go:42: 16:23:51 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.877120 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.877126 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.877132 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.877138 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.877144 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.877150 | controller | - ' ]' 2026-03-20 16:26:25.877156 | controller | - ' logger.go:42: 16:23:51 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.877162 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.877168 | controller | - ' logger.go:42: 16:23:51 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.877173 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.877179 | controller | - ' logger.go:42: 16:23:52 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.877185 | controller | - ' logger.go:42: 16:23:53 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.877191 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.877197 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.877203 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.877213 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.877219 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.877225 | controller | - ' ]' 2026-03-20 16:26:25.877230 | controller | - ' logger.go:42: 16:23:53 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.877236 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.877242 | controller | - ' logger.go:42: 16:23:53 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.877248 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.877254 | controller | - ' logger.go:42: 16:23:53 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.877260 | controller | - ' logger.go:42: 16:23:54 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.877266 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.877281 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.877287 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.877294 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.877299 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.877305 | controller | - ' ]' 2026-03-20 16:26:25.877312 | controller | - ' logger.go:42: 16:23:54 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.877319 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.877325 | controller | - ' logger.go:42: 16:23:54 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.877333 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.877340 | controller | - ' logger.go:42: 16:23:54 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.877345 | controller | - ' logger.go:42: 16:23:55 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.877351 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.877357 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.877363 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.877369 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.877375 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.877381 | controller | - ' ]' 2026-03-20 16:26:25.877387 | controller | - ' logger.go:42: 16:23:55 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.877393 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.877398 | controller | - ' logger.go:42: 16:23:55 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.877404 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.877410 | controller | - ' logger.go:42: 16:23:55 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.877416 | controller | - ' logger.go:42: 16:23:56 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.877422 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.877427 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.877436 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.877442 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.877448 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.877454 | controller | - ' ]' 2026-03-20 16:26:25.877460 | controller | - ' logger.go:42: 16:23:56 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.877466 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.877472 | controller | - ' logger.go:42: 16:23:56 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.877480 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.877486 | controller | - ' logger.go:42: 16:23:57 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.877492 | controller | - ' logger.go:42: 16:23:58 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.877498 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.877506 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.877512 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.877518 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.877524 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.877530 | controller | - ' ]' 2026-03-20 16:26:25.877535 | controller | - ' logger.go:42: 16:23:58 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.877541 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.877547 | controller | - ' logger.go:42: 16:23:58 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.877553 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.877559 | controller | - ' logger.go:42: 16:23:58 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.877565 | controller | - ' logger.go:42: 16:23:59 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.877571 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.877577 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.877583 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.877589 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.877594 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.877600 | controller | - ' ]' 2026-03-20 16:26:25.877606 | controller | - ' logger.go:42: 16:23:59 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.877612 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.877618 | controller | - ' logger.go:42: 16:23:59 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.877624 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.877630 | controller | - ' logger.go:42: 16:23:59 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.877645 | controller | - ' logger.go:42: 16:24:00 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.877651 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.877657 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.877663 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.877669 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.877675 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.877681 | controller | - ' ]' 2026-03-20 16:26:25.877687 | controller | - ' logger.go:42: 16:24:00 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.877708 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.877717 | controller | - ' logger.go:42: 16:24:00 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.877723 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.877729 | controller | - ' logger.go:42: 16:24:00 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.877735 | controller | - ' logger.go:42: 16:24:01 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.877741 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.877746 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.877755 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.877761 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.877767 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.877773 | controller | - ' ]' 2026-03-20 16:26:25.877779 | controller | - ' logger.go:42: 16:24:01 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.877785 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.877795 | controller | - ' logger.go:42: 16:24:01 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.877801 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.877807 | controller | - ' logger.go:42: 16:24:01 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.877813 | controller | - ' logger.go:42: 16:24:02 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.877819 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.877825 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.877831 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.877837 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.877843 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.877849 | controller | - ' ]' 2026-03-20 16:26:25.877855 | controller | - ' logger.go:42: 16:24:02 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.877861 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.877867 | controller | - ' logger.go:42: 16:24:02 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.877872 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.877878 | controller | - ' logger.go:42: 16:24:03 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.877884 | controller | - ' logger.go:42: 16:24:04 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.877890 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.877896 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.877902 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.877908 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.877913 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.877919 | controller | - ' ]' 2026-03-20 16:26:25.877925 | controller | - ' logger.go:42: 16:24:04 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.877931 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.877937 | controller | - ' logger.go:42: 16:24:04 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.877943 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.877949 | controller | - ' logger.go:42: 16:24:04 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.877954 | controller | - ' logger.go:42: 16:24:05 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.877960 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.877966 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.877972 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.877980 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.877986 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.877992 | controller | - ' ]' 2026-03-20 16:26:25.877998 | controller | - ' logger.go:42: 16:24:05 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.878006 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.878012 | controller | - ' logger.go:42: 16:24:05 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.878033 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.878039 | controller | - ' logger.go:42: 16:24:05 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.878045 | controller | - ' logger.go:42: 16:24:06 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.878051 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.878057 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.878063 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.878069 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.878075 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.878081 | controller | - ' ]' 2026-03-20 16:26:25.878086 | controller | - ' logger.go:42: 16:24:06 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.878092 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.878098 | controller | - ' logger.go:42: 16:24:06 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.878104 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.878110 | controller | - ' logger.go:42: 16:24:06 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.878116 | controller | - ' logger.go:42: 16:24:07 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.878122 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.878127 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.878133 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.878139 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.878144 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.878149 | controller | - ' ]' 2026-03-20 16:26:25.878155 | controller | - ' logger.go:42: 16:24:07 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.878160 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.878166 | controller | - ' logger.go:42: 16:24:07 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.878172 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.878177 | controller | - ' logger.go:42: 16:24:07 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.878184 | controller | - ' logger.go:42: 16:24:08 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.878189 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.878195 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.878201 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.878207 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.878212 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.878218 | controller | - ' ]' 2026-03-20 16:26:25.878224 | controller | - ' logger.go:42: 16:24:08 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.878229 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.878235 | controller | - ' logger.go:42: 16:24:08 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.878240 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.878246 | controller | - ' logger.go:42: 16:24:09 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.878251 | controller | - ' logger.go:42: 16:24:10 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.878261 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.878266 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.878271 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.878277 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.878282 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.878288 | controller | - ' ]' 2026-03-20 16:26:25.878294 | controller | - ' logger.go:42: 16:24:10 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.878299 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.878305 | controller | - ' logger.go:42: 16:24:10 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.878311 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.878317 | controller | - ' logger.go:42: 16:24:10 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.878322 | controller | - ' logger.go:42: 16:24:11 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.878328 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.878334 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.878344 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.878350 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.878355 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.878361 | controller | - ' ]' 2026-03-20 16:26:25.878367 | controller | - ' logger.go:42: 16:24:11 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.878373 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.878391 | controller | - ' logger.go:42: 16:24:11 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.878397 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.878403 | controller | - ' logger.go:42: 16:24:11 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.878408 | controller | - ' logger.go:42: 16:24:12 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.878414 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.878420 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.878426 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.878432 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.878436 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.878440 | controller | - ' ]' 2026-03-20 16:26:25.878447 | controller | - ' logger.go:42: 16:24:12 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.878451 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.878455 | controller | - ' logger.go:42: 16:24:12 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.878459 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.878463 | controller | - ' logger.go:42: 16:24:12 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.878467 | controller | - ' logger.go:42: 16:24:13 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.878472 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.878476 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.878480 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.878484 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.878488 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.878495 | controller | - ' ]' 2026-03-20 16:26:25.878499 | controller | - ' logger.go:42: 16:24:13 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.878503 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.878508 | controller | - ' logger.go:42: 16:24:13 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.878512 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.878518 | controller | - ' logger.go:42: 16:24:13 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.878522 | controller | - ' logger.go:42: 16:24:14 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.878526 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.878530 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.878535 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.878539 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.878543 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.878547 | controller | - ' ]' 2026-03-20 16:26:25.878551 | controller | - ' logger.go:42: 16:24:14 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.878555 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.878560 | controller | - ' logger.go:42: 16:24:14 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.878564 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.878570 | controller | - ' logger.go:42: 16:24:15 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.878575 | controller | - ' logger.go:42: 16:24:16 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.878593 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.878600 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.878606 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.878611 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.878615 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.878619 | controller | - ' ]' 2026-03-20 16:26:25.878623 | controller | - ' logger.go:42: 16:24:16 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.878627 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.878631 | controller | - ' logger.go:42: 16:24:16 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.878636 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.878640 | controller | - ' logger.go:42: 16:24:16 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.878644 | controller | - ' logger.go:42: 16:24:17 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.878648 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.878652 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.878656 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.878660 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.878664 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.878668 | controller | - ' ]' 2026-03-20 16:26:25.878684 | controller | - ' logger.go:42: 16:24:17 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.878689 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.878718 | controller | - ' logger.go:42: 16:24:17 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.878722 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.878731 | controller | - ' logger.go:42: 16:24:17 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.878736 | controller | - ' logger.go:42: 16:24:18 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.878740 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.878744 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.878748 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.878753 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.878757 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.878761 | controller | - ' ]' 2026-03-20 16:26:25.878765 | controller | - ' logger.go:42: 16:24:18 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.878769 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.878773 | controller | - ' logger.go:42: 16:24:18 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.878778 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.878782 | controller | - ' logger.go:42: 16:24:18 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.878786 | controller | - ' logger.go:42: 16:24:19 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.878790 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.878794 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.878798 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.878802 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.878807 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.878811 | controller | - ' ]' 2026-03-20 16:26:25.878815 | controller | - ' logger.go:42: 16:24:19 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.878819 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.878823 | controller | - ' logger.go:42: 16:24:19 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.878827 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.878836 | controller | - ' logger.go:42: 16:24:20 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.878840 | controller | - ' logger.go:42: 16:24:21 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.878844 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.878848 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.878852 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.878857 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.878861 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.878865 | controller | - ' ]' 2026-03-20 16:26:25.878869 | controller | - ' logger.go:42: 16:24:21 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.878875 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.878891 | controller | - ' logger.go:42: 16:24:21 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.878899 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.878906 | controller | - ' logger.go:42: 16:24:21 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.878912 | controller | - ' logger.go:42: 16:24:22 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.878918 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.878924 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.878930 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.878940 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.878947 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.878952 | controller | - ' ]' 2026-03-20 16:26:25.878957 | controller | - ' logger.go:42: 16:24:22 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.878963 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.878969 | controller | - ' logger.go:42: 16:24:22 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.878974 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.878980 | controller | - ' logger.go:42: 16:24:22 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.878986 | controller | - ' logger.go:42: 16:24:23 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.878991 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.878997 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.879003 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.879022 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.879029 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.879035 | controller | - ' ]' 2026-03-20 16:26:25.879042 | controller | - ' logger.go:42: 16:24:23 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.879048 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.879054 | controller | - ' logger.go:42: 16:24:23 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.879060 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.879066 | controller | - ' logger.go:42: 16:24:23 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.879072 | controller | - ' logger.go:42: 16:24:24 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.879077 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.879084 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.879090 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.879095 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.879101 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.879107 | controller | - ' ]' 2026-03-20 16:26:25.879113 | controller | - ' logger.go:42: 16:24:24 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.879119 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.879125 | controller | - ' logger.go:42: 16:24:24 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.879131 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.879137 | controller | - ' logger.go:42: 16:24:25 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.879143 | controller | - ' logger.go:42: 16:24:26 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.879149 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.879155 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.879165 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.879171 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.879177 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.879183 | controller | - ' ]' 2026-03-20 16:26:25.879189 | controller | - ' logger.go:42: 16:24:26 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.879196 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.879202 | controller | - ' logger.go:42: 16:24:26 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.879211 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.879217 | controller | - ' logger.go:42: 16:24:26 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.879223 | controller | - ' logger.go:42: 16:24:27 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.879229 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.879235 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.879240 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.879246 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.879251 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.879257 | controller | - ' ]' 2026-03-20 16:26:25.879263 | controller | - ' logger.go:42: 16:24:27 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.879269 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.879275 | controller | - ' logger.go:42: 16:24:27 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.879281 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.879287 | controller | - ' logger.go:42: 16:24:27 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.879305 | controller | - ' logger.go:42: 16:24:28 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.879314 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.879320 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.879326 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.879333 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.879340 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.879346 | controller | - ' ]' 2026-03-20 16:26:25.879352 | controller | - ' logger.go:42: 16:24:28 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.879358 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.879364 | controller | - ' logger.go:42: 16:24:28 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.879370 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.879376 | controller | - ' logger.go:42: 16:24:28 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.879382 | controller | - ' logger.go:42: 16:24:29 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.879386 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.879390 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.879406 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.879412 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.879416 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.879420 | controller | - ' ]' 2026-03-20 16:26:25.879424 | controller | - ' logger.go:42: 16:24:29 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.879428 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.879433 | controller | - ' logger.go:42: 16:24:29 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.879437 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.879441 | controller | - ' logger.go:42: 16:24:29 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.879445 | controller | - ' logger.go:42: 16:24:30 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.879449 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.879453 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.879460 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.879465 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.879469 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.879473 | controller | - ' ]' 2026-03-20 16:26:25.879477 | controller | - ' logger.go:42: 16:24:30 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.879481 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.879486 | controller | - ' logger.go:42: 16:24:30 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.879490 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.879494 | controller | - ' logger.go:42: 16:24:31 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.879498 | controller | - ' logger.go:42: 16:24:32 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.879502 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.879506 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.879511 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.879515 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.879519 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.879523 | controller | - ' ]' 2026-03-20 16:26:25.879527 | controller | - ' logger.go:42: 16:24:32 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.879532 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.879540 | controller | - ' logger.go:42: 16:24:32 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.879545 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.879549 | controller | - ' logger.go:42: 16:24:32 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.879553 | controller | - ' logger.go:42: 16:24:33 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.879557 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.879563 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.879569 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.879586 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.879594 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.879600 | controller | - ' ]' 2026-03-20 16:26:25.879608 | controller | - ' logger.go:42: 16:24:33 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.879612 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.879617 | controller | - ' logger.go:42: 16:24:33 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.879621 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.879625 | controller | - ' logger.go:42: 16:24:33 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.879629 | controller | - ' logger.go:42: 16:24:34 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.879634 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.879638 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.879642 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.879646 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.879650 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.879655 | controller | - ' ]' 2026-03-20 16:26:25.879659 | controller | - ' logger.go:42: 16:24:34 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.879666 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.879670 | controller | - ' logger.go:42: 16:24:34 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.879674 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.879678 | controller | - ' logger.go:42: 16:24:34 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.879687 | controller | - ' logger.go:42: 16:24:35 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.879707 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.879717 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.879722 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.879726 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.879730 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.879734 | controller | - ' ]' 2026-03-20 16:26:25.879738 | controller | - ' logger.go:42: 16:24:35 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.879743 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.879747 | controller | - ' logger.go:42: 16:24:35 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.879751 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.879755 | controller | - ' logger.go:42: 16:24:36 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.879759 | controller | - ' logger.go:42: 16:24:37 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.879763 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.879768 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.879772 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.879776 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.879780 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.879784 | controller | - ' ]' 2026-03-20 16:26:25.879788 | controller | - ' logger.go:42: 16:24:37 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.879793 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.879822 | controller | - ' logger.go:42: 16:24:37 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.879828 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.879833 | controller | - ' logger.go:42: 16:24:37 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.879837 | controller | - ' logger.go:42: 16:24:38 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.879841 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.879846 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.879850 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.879854 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.879858 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.879862 | controller | - ' ]' 2026-03-20 16:26:25.879867 | controller | - ' logger.go:42: 16:24:38 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.879871 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.879875 | controller | - ' logger.go:42: 16:24:38 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.879879 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.879883 | controller | - ' logger.go:42: 16:24:38 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.879887 | controller | - ' logger.go:42: 16:24:39 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.879894 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.879898 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.879902 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.879906 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.879910 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.879915 | controller | - ' ]' 2026-03-20 16:26:25.879919 | controller | - ' logger.go:42: 16:24:39 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.879923 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.879927 | controller | - ' logger.go:42: 16:24:39 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.879932 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.879936 | controller | - ' logger.go:42: 16:24:39 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.879940 | controller | - ' logger.go:42: 16:24:40 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.879944 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.879948 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.879955 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.879959 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.879963 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.879968 | controller | - ' ]' 2026-03-20 16:26:25.879972 | controller | - ' logger.go:42: 16:24:40 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.879976 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.879980 | controller | - ' logger.go:42: 16:24:40 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.879984 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.879993 | controller | - ' logger.go:42: 16:24:41 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.879997 | controller | - ' logger.go:42: 16:24:42 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.880002 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.880006 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.880010 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.880014 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.880018 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.880023 | controller | - ' ]' 2026-03-20 16:26:25.880027 | controller | - ' logger.go:42: 16:24:42 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.880031 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.880035 | controller | - ' logger.go:42: 16:24:42 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.880039 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.880044 | controller | - ' logger.go:42: 16:24:42 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.880048 | controller | - ' logger.go:42: 16:24:43 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.880052 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.880056 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.880060 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.880066 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.880071 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.880078 | controller | - ' ]' 2026-03-20 16:26:25.880082 | controller | - ' logger.go:42: 16:24:43 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.880086 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.880090 | controller | - ' logger.go:42: 16:24:43 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.880094 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.880099 | controller | - ' logger.go:42: 16:24:43 | watcher-cinder/2-deploy-cinder | + ''['' 2 == 0 '']''' 2026-03-20 16:26:25.880103 | controller | - ' logger.go:42: 16:24:44 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.880107 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.880111 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.880116 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.880120 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.880124 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.880128 | controller | - ' ]' 2026-03-20 16:26:25.880132 | controller | - ' logger.go:42: 16:24:44 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.880137 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.880141 | controller | - ' logger.go:42: 16:24:44 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.880145 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.880149 | controller | - ' logger.go:42: 16:24:44 | watcher-cinder/2-deploy-cinder | Error from server 2026-03-20 16:26:25.880155 | controller | (BadRequest): container "watcher-decision-engine" in pod "watcher-kuttl-decision-engine-0" 2026-03-20 16:26:25.880159 | controller | is waiting to start: ContainerCreating' 2026-03-20 16:26:25.880163 | controller | - ' logger.go:42: 16:24:44 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']''' 2026-03-20 16:26:25.880168 | controller | - ' logger.go:42: 16:24:45 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.880172 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.880176 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.880180 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.880184 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.880188 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.880193 | controller | - ' ]' 2026-03-20 16:26:25.880197 | controller | - ' logger.go:42: 16:24:45 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.880201 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.880205 | controller | - ' logger.go:42: 16:24:45 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.880210 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.880214 | controller | - ' logger.go:42: 16:24:45 | watcher-cinder/2-deploy-cinder | Error from server 2026-03-20 16:26:25.880218 | controller | (BadRequest): container "watcher-decision-engine" in pod "watcher-kuttl-decision-engine-0" 2026-03-20 16:26:25.880222 | controller | is waiting to start: ContainerCreating' 2026-03-20 16:26:25.880226 | controller | - ' logger.go:42: 16:24:45 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']''' 2026-03-20 16:26:25.880230 | controller | - ' logger.go:42: 16:24:46 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.880235 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.880241 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.880246 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.880250 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.880256 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.880263 | controller | - ' ]' 2026-03-20 16:26:25.880267 | controller | - ' logger.go:42: 16:24:46 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.880271 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.880275 | controller | - ' logger.go:42: 16:24:46 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.880280 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.880284 | controller | - ' logger.go:42: 16:24:46 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']''' 2026-03-20 16:26:25.880288 | controller | - ' logger.go:42: 16:24:47 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.880292 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.880296 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.880300 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.880305 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.880309 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.880313 | controller | - ' ]' 2026-03-20 16:26:25.880317 | controller | - ' logger.go:42: 16:24:47 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.880321 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.880326 | controller | - ' logger.go:42: 16:24:47 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.880330 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.880334 | controller | - ' logger.go:42: 16:24:48 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']''' 2026-03-20 16:26:25.880338 | controller | - ' logger.go:42: 16:24:49 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.880342 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.880346 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.880351 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.880355 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.880359 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.880363 | controller | - ' ]' 2026-03-20 16:26:25.880367 | controller | - ' logger.go:42: 16:24:49 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.880372 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.880376 | controller | - ' logger.go:42: 16:24:49 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.880380 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.880384 | controller | - ' logger.go:42: 16:24:49 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']''' 2026-03-20 16:26:25.880388 | controller | - ' logger.go:42: 16:24:50 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.880392 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.880398 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.880402 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.880407 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.880411 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.880415 | controller | - ' ]' 2026-03-20 16:26:25.880419 | controller | - ' logger.go:42: 16:24:50 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.880423 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.880427 | controller | - ' logger.go:42: 16:24:50 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.880434 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.880438 | controller | - ' logger.go:42: 16:24:50 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']''' 2026-03-20 16:26:25.880443 | controller | - ' logger.go:42: 16:24:51 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.880447 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.880451 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.880455 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.880460 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.880464 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.880468 | controller | - ' ]' 2026-03-20 16:26:25.880472 | controller | - ' logger.go:42: 16:24:51 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.880476 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.880480 | controller | - ' logger.go:42: 16:24:51 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.880485 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.880489 | controller | - ' logger.go:42: 16:24:51 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']''' 2026-03-20 16:26:25.880495 | controller | - ' logger.go:42: 16:24:52 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.880500 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.880504 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.880508 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.880512 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.880516 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.880521 | controller | - ' ]' 2026-03-20 16:26:25.880525 | controller | - ' logger.go:42: 16:24:52 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.880529 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.880533 | controller | - ' logger.go:42: 16:24:52 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.880537 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.880542 | controller | - ' logger.go:42: 16:24:52 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']''' 2026-03-20 16:26:25.880546 | controller | - ' logger.go:42: 16:24:53 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.880550 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.880554 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.880558 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.880564 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.880570 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.880576 | controller | - ' ]' 2026-03-20 16:26:25.880582 | controller | - ' logger.go:42: 16:24:53 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.880588 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.880594 | controller | - ' logger.go:42: 16:24:53 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.880598 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.880602 | controller | - ' logger.go:42: 16:24:54 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']''' 2026-03-20 16:26:25.880607 | controller | - ' logger.go:42: 16:24:55 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.880611 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.880615 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.880623 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.880627 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.880631 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.880636 | controller | - ' ]' 2026-03-20 16:26:25.880640 | controller | - ' logger.go:42: 16:24:55 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.880644 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.880648 | controller | - ' logger.go:42: 16:24:55 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.880652 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.880657 | controller | - ' logger.go:42: 16:24:55 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']''' 2026-03-20 16:26:25.880661 | controller | - ' logger.go:42: 16:24:56 | watcher-cinder/2-deploy-cinder | running command: 2026-03-20 16:26:25.880665 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.880669 | controller | - ' # check that the decision detects that there is a cinder service and' 2026-03-20 16:26:25.880673 | controller | - ' # does not log that storage collector is skipped' 2026-03-20 16:26:25.880677 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.880682 | controller | storage service is not enabled, skipping storage collector'')" == 0 ]' 2026-03-20 16:26:25.880686 | controller | - ' ]' 2026-03-20 16:26:25.880690 | controller | - ' logger.go:42: 16:24:56 | watcher-cinder/2-deploy-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.880711 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.880716 | controller | - ' logger.go:42: 16:24:56 | watcher-cinder/2-deploy-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.880720 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.880724 | controller | - ' logger.go:42: 16:24:56 | watcher-cinder/2-deploy-cinder | + ''['' 0 == 0 '']''' 2026-03-20 16:26:25.880728 | controller | - ' logger.go:42: 16:24:56 | watcher-cinder/2-deploy-cinder | test step completed 2026-03-20 16:26:25.880732 | controller | 2-deploy-cinder' 2026-03-20 16:26:25.880737 | controller | - ' logger.go:42: 16:24:56 | watcher-cinder/3-remove-cinder | starting test step 2026-03-20 16:26:25.880741 | controller | 3-remove-cinder' 2026-03-20 16:26:25.880745 | controller | - ' logger.go:42: 16:24:56 | watcher-cinder/3-remove-cinder | OpenStackControlPlane:watcher-kuttl-default/openstack 2026-03-20 16:26:25.880749 | controller | updated' 2026-03-20 16:26:25.880757 | controller | - ' logger.go:42: 16:24:56 | watcher-cinder/3-remove-cinder | running command: 2026-03-20 16:26:25.880775 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.880779 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.880783 | controller | service' 2026-03-20 16:26:25.880787 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.880792 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.880800 | controller | - ' ]' 2026-03-20 16:26:25.880805 | controller | - ' logger.go:42: 16:24:56 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.880809 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.880813 | controller | - ' logger.go:42: 16:24:56 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.880817 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.880822 | controller | - ' logger.go:42: 16:24:56 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']''' 2026-03-20 16:26:25.880826 | controller | - ' logger.go:42: 16:24:57 | watcher-cinder/3-remove-cinder | running command: 2026-03-20 16:26:25.880830 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.880834 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.880840 | controller | service' 2026-03-20 16:26:25.880845 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.880849 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.880853 | controller | - ' ]' 2026-03-20 16:26:25.880857 | controller | - ' logger.go:42: 16:24:57 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.880861 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.880866 | controller | - ' logger.go:42: 16:24:57 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.880870 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.880874 | controller | - ' logger.go:42: 16:24:58 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']''' 2026-03-20 16:26:25.880878 | controller | - ' logger.go:42: 16:24:59 | watcher-cinder/3-remove-cinder | running command: 2026-03-20 16:26:25.880882 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.880886 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.880890 | controller | service' 2026-03-20 16:26:25.880895 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.880899 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.880903 | controller | - ' ]' 2026-03-20 16:26:25.880907 | controller | - ' logger.go:42: 16:24:59 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.880911 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.880915 | controller | - ' logger.go:42: 16:24:59 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.880920 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.880924 | controller | - ' logger.go:42: 16:24:59 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']''' 2026-03-20 16:26:25.880928 | controller | - ' logger.go:42: 16:25:00 | watcher-cinder/3-remove-cinder | running command: 2026-03-20 16:26:25.880932 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.880936 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.880940 | controller | service' 2026-03-20 16:26:25.880945 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.880949 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.880953 | controller | - ' ]' 2026-03-20 16:26:25.880957 | controller | - ' logger.go:42: 16:25:00 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.880961 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.880966 | controller | - ' logger.go:42: 16:25:00 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.880970 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.880974 | controller | - ' logger.go:42: 16:25:00 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']''' 2026-03-20 16:26:25.880978 | controller | - ' logger.go:42: 16:25:01 | watcher-cinder/3-remove-cinder | running command: 2026-03-20 16:26:25.880983 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.880987 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.880991 | controller | service' 2026-03-20 16:26:25.880996 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.881000 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.881004 | controller | - ' ]' 2026-03-20 16:26:25.881009 | controller | - ' logger.go:42: 16:25:01 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.881013 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.881017 | controller | - ' logger.go:42: 16:25:01 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.881024 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.881030 | controller | - ' logger.go:42: 16:25:01 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']''' 2026-03-20 16:26:25.881034 | controller | - ' logger.go:42: 16:25:02 | watcher-cinder/3-remove-cinder | running command: 2026-03-20 16:26:25.881038 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.881043 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.881047 | controller | service' 2026-03-20 16:26:25.881051 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.881055 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.881059 | controller | - ' ]' 2026-03-20 16:26:25.881066 | controller | - ' logger.go:42: 16:25:02 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.881070 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.881075 | controller | - ' logger.go:42: 16:25:02 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.881079 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.881083 | controller | - ' logger.go:42: 16:25:02 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']''' 2026-03-20 16:26:25.881087 | controller | - ' logger.go:42: 16:25:03 | watcher-cinder/3-remove-cinder | running command: 2026-03-20 16:26:25.881091 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.881096 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.881102 | controller | service' 2026-03-20 16:26:25.881107 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.881111 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.881115 | controller | - ' ]' 2026-03-20 16:26:25.881119 | controller | - ' logger.go:42: 16:25:03 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.881124 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.881128 | controller | - ' logger.go:42: 16:25:03 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.881132 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.881136 | controller | - ' logger.go:42: 16:25:04 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']''' 2026-03-20 16:26:25.881140 | controller | - ' logger.go:42: 16:25:05 | watcher-cinder/3-remove-cinder | running command: 2026-03-20 16:26:25.881145 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.881149 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.881153 | controller | service' 2026-03-20 16:26:25.881157 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.881161 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.881166 | controller | - ' ]' 2026-03-20 16:26:25.881170 | controller | - ' logger.go:42: 16:25:05 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.881174 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.881178 | controller | - ' logger.go:42: 16:25:05 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.881182 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.881186 | controller | - ' logger.go:42: 16:25:05 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']''' 2026-03-20 16:26:25.881191 | controller | - ' logger.go:42: 16:25:06 | watcher-cinder/3-remove-cinder | running command: 2026-03-20 16:26:25.881195 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.881199 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.881203 | controller | service' 2026-03-20 16:26:25.881207 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.881213 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.881217 | controller | - ' ]' 2026-03-20 16:26:25.881221 | controller | - ' logger.go:42: 16:25:06 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.881226 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.881230 | controller | - ' logger.go:42: 16:25:06 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.881234 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.881238 | controller | - ' logger.go:42: 16:25:06 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']''' 2026-03-20 16:26:25.881242 | controller | - ' logger.go:42: 16:25:07 | watcher-cinder/3-remove-cinder | running command: 2026-03-20 16:26:25.881246 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.881251 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.881255 | controller | service' 2026-03-20 16:26:25.881259 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.881263 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.881267 | controller | - ' ]' 2026-03-20 16:26:25.881272 | controller | - ' logger.go:42: 16:25:07 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.881276 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.881280 | controller | - ' logger.go:42: 16:25:07 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.881284 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.881288 | controller | - ' logger.go:42: 16:25:07 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']''' 2026-03-20 16:26:25.881292 | controller | - ' logger.go:42: 16:25:08 | watcher-cinder/3-remove-cinder | running command: 2026-03-20 16:26:25.881296 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.881301 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.881305 | controller | service' 2026-03-20 16:26:25.881309 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.881313 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.881318 | controller | - ' ]' 2026-03-20 16:26:25.881322 | controller | - ' logger.go:42: 16:25:08 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.881326 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.881333 | controller | - ' logger.go:42: 16:25:08 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.881338 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.881342 | controller | - ' logger.go:42: 16:25:08 | watcher-cinder/3-remove-cinder | Error from server 2026-03-20 16:26:25.881346 | controller | (BadRequest): container "watcher-decision-engine" in pod "watcher-kuttl-decision-engine-0" 2026-03-20 16:26:25.881351 | controller | is waiting to start: ContainerCreating' 2026-03-20 16:26:25.881355 | controller | - ' logger.go:42: 16:25:08 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']''' 2026-03-20 16:26:25.881359 | controller | - ' logger.go:42: 16:25:09 | watcher-cinder/3-remove-cinder | running command: 2026-03-20 16:26:25.881478 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.881485 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.881490 | controller | service' 2026-03-20 16:26:25.881494 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.881498 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.881502 | controller | - ' ]' 2026-03-20 16:26:25.881506 | controller | - ' logger.go:42: 16:25:09 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.881511 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.881517 | controller | - ' logger.go:42: 16:25:09 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.881521 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.881525 | controller | - ' logger.go:42: 16:25:09 | watcher-cinder/3-remove-cinder | Error from server 2026-03-20 16:26:25.881530 | controller | (BadRequest): container "watcher-decision-engine" in pod "watcher-kuttl-decision-engine-0" 2026-03-20 16:26:25.881534 | controller | is waiting to start: ContainerCreating' 2026-03-20 16:26:25.881538 | controller | - ' logger.go:42: 16:25:09 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']''' 2026-03-20 16:26:25.881542 | controller | - ' logger.go:42: 16:25:10 | watcher-cinder/3-remove-cinder | running command: 2026-03-20 16:26:25.881549 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.881553 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.881557 | controller | service' 2026-03-20 16:26:25.881563 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.881569 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.881575 | controller | - ' ]' 2026-03-20 16:26:25.881581 | controller | - ' logger.go:42: 16:25:10 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.881587 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.881593 | controller | - ' logger.go:42: 16:25:10 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.881599 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.881604 | controller | - ' logger.go:42: 16:25:11 | watcher-cinder/3-remove-cinder | + ''['' 0 == 2 '']''' 2026-03-20 16:26:25.881608 | controller | - ' logger.go:42: 16:25:12 | watcher-cinder/3-remove-cinder | running command: 2026-03-20 16:26:25.881612 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.881617 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.881621 | controller | service' 2026-03-20 16:26:25.881625 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.881629 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.881633 | controller | - ' ]' 2026-03-20 16:26:25.881637 | controller | - ' logger.go:42: 16:25:12 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.881642 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.881646 | controller | - ' logger.go:42: 16:25:12 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.881650 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.881654 | controller | - ' logger.go:42: 16:25:12 | watcher-cinder/3-remove-cinder | + ''['' 1 == 2 '']''' 2026-03-20 16:26:25.881658 | controller | - ' logger.go:42: 16:25:13 | watcher-cinder/3-remove-cinder | running command: 2026-03-20 16:26:25.881662 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.881667 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.881671 | controller | service' 2026-03-20 16:26:25.881675 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.881679 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.881683 | controller | - ' ]' 2026-03-20 16:26:25.881687 | controller | - ' logger.go:42: 16:25:13 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.881711 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.881718 | controller | - ' logger.go:42: 16:25:13 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.881723 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.881727 | controller | - ' logger.go:42: 16:25:13 | watcher-cinder/3-remove-cinder | + ''['' 2 == 2 '']''' 2026-03-20 16:26:25.881733 | controller | - ' logger.go:42: 16:25:14 | watcher-cinder/3-remove-cinder | running command: 2026-03-20 16:26:25.881738 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.881742 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.881751 | controller | service' 2026-03-20 16:26:25.881755 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.881760 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.881764 | controller | - ' ]' 2026-03-20 16:26:25.881768 | controller | - ' logger.go:42: 16:25:14 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.881772 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.881777 | controller | - ' logger.go:42: 16:25:14 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.881784 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.881788 | controller | - ' logger.go:42: 16:25:14 | watcher-cinder/3-remove-cinder | + ''['' 2 == 2 '']''' 2026-03-20 16:26:25.881792 | controller | - ' logger.go:42: 16:25:15 | watcher-cinder/3-remove-cinder | running command: 2026-03-20 16:26:25.881796 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.881800 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.881805 | controller | service' 2026-03-20 16:26:25.881809 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.881814 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.881823 | controller | - ' ]' 2026-03-20 16:26:25.881837 | controller | - ' logger.go:42: 16:25:15 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.881844 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.881850 | controller | - ' logger.go:42: 16:25:15 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.881856 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.881862 | controller | - ' logger.go:42: 16:25:15 | watcher-cinder/3-remove-cinder | + ''['' 2 == 2 '']''' 2026-03-20 16:26:25.881868 | controller | - ' logger.go:42: 16:25:16 | watcher-cinder/3-remove-cinder | running command: 2026-03-20 16:26:25.881873 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.881879 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.881885 | controller | service' 2026-03-20 16:26:25.881891 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.881896 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.881902 | controller | - ' ]' 2026-03-20 16:26:25.881908 | controller | - ' logger.go:42: 16:25:16 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.881914 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.881920 | controller | - ' logger.go:42: 16:25:16 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.881931 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.881937 | controller | - ' logger.go:42: 16:25:17 | watcher-cinder/3-remove-cinder | + ''['' 2 == 2 '']''' 2026-03-20 16:26:25.881943 | controller | - ' logger.go:42: 16:25:18 | watcher-cinder/3-remove-cinder | running command: 2026-03-20 16:26:25.881949 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.881955 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.881963 | controller | service' 2026-03-20 16:26:25.881969 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.881975 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.881981 | controller | - ' ]' 2026-03-20 16:26:25.881990 | controller | - ' logger.go:42: 16:25:18 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.881997 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.882003 | controller | - ' logger.go:42: 16:25:18 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.882009 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.882015 | controller | - ' logger.go:42: 16:25:18 | watcher-cinder/3-remove-cinder | + ''['' 2 == 2 '']''' 2026-03-20 16:26:25.882021 | controller | - ' logger.go:42: 16:25:19 | watcher-cinder/3-remove-cinder | running command: 2026-03-20 16:26:25.882027 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.882033 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.882039 | controller | service' 2026-03-20 16:26:25.882045 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.882051 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.882056 | controller | - ' ]' 2026-03-20 16:26:25.882062 | controller | - ' logger.go:42: 16:25:19 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.882066 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.882070 | controller | - ' logger.go:42: 16:25:19 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.882074 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.882078 | controller | - ' logger.go:42: 16:25:19 | watcher-cinder/3-remove-cinder | + ''['' 2 == 2 '']''' 2026-03-20 16:26:25.882082 | controller | - ' logger.go:42: 16:25:20 | watcher-cinder/3-remove-cinder | running command: 2026-03-20 16:26:25.882086 | controller | [sh -c set -euxo pipefail' 2026-03-20 16:26:25.882091 | controller | - ' # check that the decision engine correctly detects that there is no cinder 2026-03-20 16:26:25.882127 | controller | service' 2026-03-20 16:26:25.882137 | controller | - ' [ "$(oc logs -n $NAMESPACE watcher-kuttl-decision-engine-0 |grep -c ''Block 2026-03-20 16:26:25.882141 | controller | storage service is not enabled, skipping storage collector'')" == 2 ]' 2026-03-20 16:26:25.882145 | controller | - ' ]' 2026-03-20 16:26:25.882150 | controller | - ' logger.go:42: 16:25:20 | watcher-cinder/3-remove-cinder | ++ oc logs -n watcher-kuttl-default 2026-03-20 16:26:25.882154 | controller | watcher-kuttl-decision-engine-0' 2026-03-20 16:26:25.882158 | controller | - ' logger.go:42: 16:25:20 | watcher-cinder/3-remove-cinder | ++ grep -c ''Block 2026-03-20 16:26:25.882162 | controller | storage service is not enabled, skipping storage collector''' 2026-03-20 16:26:25.882166 | controller | - ' logger.go:42: 16:25:20 | watcher-cinder/3-remove-cinder | + ''['' 2 == 2 '']''' 2026-03-20 16:26:25.882171 | controller | - ' logger.go:42: 16:25:20 | watcher-cinder/3-remove-cinder | test step completed 2026-03-20 16:26:25.882175 | controller | 3-remove-cinder' 2026-03-20 16:26:25.882179 | controller | - ' logger.go:42: 16:25:20 | watcher-cinder/4-cleanup-watcher | starting test step 2026-03-20 16:26:25.882183 | controller | 4-cleanup-watcher' 2026-03-20 16:26:25.882187 | controller | - ' logger.go:42: 16:25:27 | watcher-cinder/4-cleanup-watcher | test step completed 2026-03-20 16:26:25.882191 | controller | 4-cleanup-watcher' 2026-03-20 16:26:25.882196 | controller | - ' logger.go:42: 16:25:27 | watcher-cinder | skipping kubernetes event logging' 2026-03-20 16:26:25.882200 | controller | - === CONT kuttl/harness/watcher-api-scaling 2026-03-20 16:26:25.882204 | controller | - ' logger.go:42: 16:25:27 | watcher-api-scaling | Skipping creation of user-supplied 2026-03-20 16:26:25.882208 | controller | namespace: watcher-kuttl-default' 2026-03-20 16:26:25.882212 | controller | - ' logger.go:42: 16:25:27 | watcher-api-scaling/0-cleanup-watcher | starting test 2026-03-20 16:26:25.882217 | controller | step 0-cleanup-watcher' 2026-03-20 16:26:25.882221 | controller | - ' logger.go:42: 16:25:27 | watcher-api-scaling/0-cleanup-watcher | test step 2026-03-20 16:26:25.882225 | controller | completed 0-cleanup-watcher' 2026-03-20 16:26:25.882232 | controller | - ' logger.go:42: 16:25:27 | watcher-api-scaling/1-deploy-with-defaults | starting 2026-03-20 16:26:25.882236 | controller | test step 1-deploy-with-defaults' 2026-03-20 16:26:25.882240 | controller | - ' logger.go:42: 16:25:27 | watcher-api-scaling/1-deploy-with-defaults | Watcher:watcher-kuttl-default/watcher-kuttl 2026-03-20 16:26:25.882244 | controller | created' 2026-03-20 16:26:25.882248 | controller | - ' logger.go:42: 16:25:28 | watcher-api-scaling/1-deploy-with-defaults | running 2026-03-20 16:26:25.882253 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.882257 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.882261 | controller | -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-20 16:26:25.882265 | controller | - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack 2026-03-20 16:26:25.882269 | controller | service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-20 16:26:25.882273 | controller | - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.882278 | controller | == ${SERVICEID} ]' 2026-03-20 16:26:25.882282 | controller | - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-20 16:26:25.882286 | controller | ]' 2026-03-20 16:26:25.882291 | controller | - ' # If we are running the container locally, skip following test' 2026-03-20 16:26:25.882295 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-20 16:26:25.882299 | controller | == "" ]; then' 2026-03-20 16:26:25.882303 | controller | - ' exit 0' 2026-03-20 16:26:25.882307 | controller | - ' fi' 2026-03-20 16:26:25.882311 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-20 16:26:25.882316 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-20 16:26:25.882320 | controller | - ' counter=0' 2026-03-20 16:26:25.882324 | controller | - ' for i in ${env_variables}; do' 2026-03-20 16:26:25.882328 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-20 16:26:25.882332 | controller | - ' echo ${i}' 2026-03-20 16:26:25.882337 | controller | - ' counter=$((counter + 1))' 2026-03-20 16:26:25.882341 | controller | - ' fi' 2026-03-20 16:26:25.882345 | controller | - ' done' 2026-03-20 16:26:25.882349 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-20 16:26:25.882353 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-20 16:26:25.882358 | controller | - ' exit 1' 2026-03-20 16:26:25.882362 | controller | - ' else' 2026-03-20 16:26:25.882366 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-20 16:26:25.882370 | controller | - ' fi' 2026-03-20 16:26:25.882374 | controller | - ' ]' 2026-03-20 16:26:25.882378 | controller | - ' logger.go:42: 16:25:28 | watcher-api-scaling/1-deploy-with-defaults | + oc 2026-03-20 16:26:25.882383 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-20 16:26:25.882387 | controller | -c Name -c Type' 2026-03-20 16:26:25.882391 | controller | - ' logger.go:42: 16:25:28 | watcher-api-scaling/1-deploy-with-defaults | ++ grep 2026-03-20 16:26:25.882395 | controller | -c ''^watcher''' 2026-03-20 16:26:25.882399 | controller | - ' logger.go:42: 16:25:30 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-20 16:26:25.882403 | controller | 0 == 1 '']''' 2026-03-20 16:26:25.882412 | controller | - ' logger.go:42: 16:25:31 | watcher-api-scaling/1-deploy-with-defaults | running 2026-03-20 16:26:25.882417 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.882421 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.882425 | controller | -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-20 16:26:25.882429 | controller | - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack 2026-03-20 16:26:25.882438 | controller | service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-20 16:26:25.882442 | controller | - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.882446 | controller | == ${SERVICEID} ]' 2026-03-20 16:26:25.882450 | controller | - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-20 16:26:25.882455 | controller | ]' 2026-03-20 16:26:25.882459 | controller | - ' # If we are running the container locally, skip following test' 2026-03-20 16:26:25.882463 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-20 16:26:25.882467 | controller | == "" ]; then' 2026-03-20 16:26:25.882471 | controller | - ' exit 0' 2026-03-20 16:26:25.882477 | controller | - ' fi' 2026-03-20 16:26:25.882481 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-20 16:26:25.882486 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-20 16:26:25.882490 | controller | - ' counter=0' 2026-03-20 16:26:25.882494 | controller | - ' for i in ${env_variables}; do' 2026-03-20 16:26:25.882498 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-20 16:26:25.882502 | controller | - ' echo ${i}' 2026-03-20 16:26:25.882507 | controller | - ' counter=$((counter + 1))' 2026-03-20 16:26:25.882511 | controller | - ' fi' 2026-03-20 16:26:25.882515 | controller | - ' done' 2026-03-20 16:26:25.882519 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-20 16:26:25.882524 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-20 16:26:25.882528 | controller | - ' exit 1' 2026-03-20 16:26:25.882532 | controller | - ' else' 2026-03-20 16:26:25.882536 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-20 16:26:25.882540 | controller | - ' fi' 2026-03-20 16:26:25.882544 | controller | - ' ]' 2026-03-20 16:26:25.882548 | controller | - ' logger.go:42: 16:25:31 | watcher-api-scaling/1-deploy-with-defaults | + oc 2026-03-20 16:26:25.882553 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-20 16:26:25.882557 | controller | -c Name -c Type' 2026-03-20 16:26:25.882561 | controller | - ' logger.go:42: 16:25:31 | watcher-api-scaling/1-deploy-with-defaults | ++ grep 2026-03-20 16:26:25.882565 | controller | -c ''^watcher''' 2026-03-20 16:26:25.882569 | controller | - ' logger.go:42: 16:25:33 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-20 16:26:25.882573 | controller | 0 == 1 '']''' 2026-03-20 16:26:25.882578 | controller | - ' logger.go:42: 16:25:34 | watcher-api-scaling/1-deploy-with-defaults | running 2026-03-20 16:26:25.882582 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.882586 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.882590 | controller | -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-20 16:26:25.882594 | controller | - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack 2026-03-20 16:26:25.882598 | controller | service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-20 16:26:25.882602 | controller | - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.882607 | controller | == ${SERVICEID} ]' 2026-03-20 16:26:25.882611 | controller | - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-20 16:26:25.882615 | controller | ]' 2026-03-20 16:26:25.882619 | controller | - ' # If we are running the container locally, skip following test' 2026-03-20 16:26:25.882623 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-20 16:26:25.882628 | controller | == "" ]; then' 2026-03-20 16:26:25.882632 | controller | - ' exit 0' 2026-03-20 16:26:25.882636 | controller | - ' fi' 2026-03-20 16:26:25.882642 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-20 16:26:25.882646 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-20 16:26:25.882650 | controller | - ' counter=0' 2026-03-20 16:26:25.882654 | controller | - ' for i in ${env_variables}; do' 2026-03-20 16:26:25.882659 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-20 16:26:25.882663 | controller | - ' echo ${i}' 2026-03-20 16:26:25.882669 | controller | - ' counter=$((counter + 1))' 2026-03-20 16:26:25.882673 | controller | - ' fi' 2026-03-20 16:26:25.882677 | controller | - ' done' 2026-03-20 16:26:25.882682 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-20 16:26:25.882686 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-20 16:26:25.882690 | controller | - ' exit 1' 2026-03-20 16:26:25.882715 | controller | - ' else' 2026-03-20 16:26:25.882721 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-20 16:26:25.882726 | controller | - ' fi' 2026-03-20 16:26:25.882730 | controller | - ' ]' 2026-03-20 16:26:25.882734 | controller | - ' logger.go:42: 16:25:34 | watcher-api-scaling/1-deploy-with-defaults | + oc 2026-03-20 16:26:25.882738 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-20 16:26:25.882742 | controller | -c Name -c Type' 2026-03-20 16:26:25.882747 | controller | - ' logger.go:42: 16:25:34 | watcher-api-scaling/1-deploy-with-defaults | ++ grep 2026-03-20 16:26:25.882751 | controller | -c ''^watcher''' 2026-03-20 16:26:25.882755 | controller | - ' logger.go:42: 16:25:36 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-20 16:26:25.882759 | controller | 1 == 1 '']''' 2026-03-20 16:26:25.882763 | controller | - ' logger.go:42: 16:25:36 | watcher-api-scaling/1-deploy-with-defaults | ++ grep 2026-03-20 16:26:25.882767 | controller | watcher' 2026-03-20 16:26:25.882772 | controller | - ' logger.go:42: 16:25:36 | watcher-api-scaling/1-deploy-with-defaults | ++ oc 2026-03-20 16:26:25.882779 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-20 16:26:25.882784 | controller | -c Name -c Type -c ID' 2026-03-20 16:26:25.882789 | controller | - ' logger.go:42: 16:25:36 | watcher-api-scaling/1-deploy-with-defaults | ++ awk 2026-03-20 16:26:25.882793 | controller | ''{print $1}''' 2026-03-20 16:26:25.882797 | controller | - ' logger.go:42: 16:25:38 | watcher-api-scaling/1-deploy-with-defaults | + SERVICEID=fc1daa96ef8c455ca09bffb24ea64ff6' 2026-03-20 16:26:25.882801 | controller | - ' logger.go:42: 16:25:38 | watcher-api-scaling/1-deploy-with-defaults | ++ oc 2026-03-20 16:26:25.882806 | controller | get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-20 16:26:25.882810 | controller | - ' logger.go:42: 16:25:38 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-20 16:26:25.882814 | controller | fc1daa96ef8c455ca09bffb24ea64ff6 == fc1daa96ef8c455ca09bffb24ea64ff6 '']''' 2026-03-20 16:26:25.882818 | controller | - ' logger.go:42: 16:25:38 | watcher-api-scaling/1-deploy-with-defaults | ++ oc 2026-03-20 16:26:25.882822 | controller | get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-20 16:26:25.882826 | controller | - ' logger.go:42: 16:25:38 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-20 16:26:25.882830 | controller | -n '''' '']''' 2026-03-20 16:26:25.882835 | controller | - ' logger.go:42: 16:25:39 | watcher-api-scaling/1-deploy-with-defaults | running 2026-03-20 16:26:25.882839 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.882843 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.882847 | controller | -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-20 16:26:25.882851 | controller | - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack 2026-03-20 16:26:25.882855 | controller | service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-20 16:26:25.882861 | controller | - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.882866 | controller | == ${SERVICEID} ]' 2026-03-20 16:26:25.882870 | controller | - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-20 16:26:25.882874 | controller | ]' 2026-03-20 16:26:25.882878 | controller | - ' # If we are running the container locally, skip following test' 2026-03-20 16:26:25.882882 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-20 16:26:25.882887 | controller | == "" ]; then' 2026-03-20 16:26:25.882891 | controller | - ' exit 0' 2026-03-20 16:26:25.882895 | controller | - ' fi' 2026-03-20 16:26:25.882901 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-20 16:26:25.882905 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-20 16:26:25.882909 | controller | - ' counter=0' 2026-03-20 16:26:25.882914 | controller | - ' for i in ${env_variables}; do' 2026-03-20 16:26:25.882918 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-20 16:26:25.882922 | controller | - ' echo ${i}' 2026-03-20 16:26:25.882926 | controller | - ' counter=$((counter + 1))' 2026-03-20 16:26:25.882930 | controller | - ' fi' 2026-03-20 16:26:25.882934 | controller | - ' done' 2026-03-20 16:26:25.882939 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-20 16:26:25.882943 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-20 16:26:25.882947 | controller | - ' exit 1' 2026-03-20 16:26:25.882951 | controller | - ' else' 2026-03-20 16:26:25.882955 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-20 16:26:25.882959 | controller | - ' fi' 2026-03-20 16:26:25.882964 | controller | - ' ]' 2026-03-20 16:26:25.882968 | controller | - ' logger.go:42: 16:25:39 | watcher-api-scaling/1-deploy-with-defaults | + oc 2026-03-20 16:26:25.882972 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-20 16:26:25.882976 | controller | -c Name -c Type' 2026-03-20 16:26:25.882980 | controller | - ' logger.go:42: 16:25:39 | watcher-api-scaling/1-deploy-with-defaults | ++ grep 2026-03-20 16:26:25.882984 | controller | -c ''^watcher''' 2026-03-20 16:26:25.882989 | controller | - ' logger.go:42: 16:25:42 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-20 16:26:25.882993 | controller | 1 == 1 '']''' 2026-03-20 16:26:25.882997 | controller | - ' logger.go:42: 16:25:42 | watcher-api-scaling/1-deploy-with-defaults | ++ oc 2026-03-20 16:26:25.883001 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-20 16:26:25.883005 | controller | -c Name -c Type -c ID' 2026-03-20 16:26:25.883009 | controller | - ' logger.go:42: 16:25:42 | watcher-api-scaling/1-deploy-with-defaults | ++ awk 2026-03-20 16:26:25.883013 | controller | ''{print $1}''' 2026-03-20 16:26:25.883018 | controller | - ' logger.go:42: 16:25:42 | watcher-api-scaling/1-deploy-with-defaults | ++ grep 2026-03-20 16:26:25.883022 | controller | watcher' 2026-03-20 16:26:25.883026 | controller | - ' logger.go:42: 16:25:44 | watcher-api-scaling/1-deploy-with-defaults | + SERVICEID=fc1daa96ef8c455ca09bffb24ea64ff6' 2026-03-20 16:26:25.883030 | controller | - ' logger.go:42: 16:25:44 | watcher-api-scaling/1-deploy-with-defaults | ++ oc 2026-03-20 16:26:25.883034 | controller | get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-20 16:26:25.883038 | controller | - ' logger.go:42: 16:25:44 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-20 16:26:25.883042 | controller | fc1daa96ef8c455ca09bffb24ea64ff6 == fc1daa96ef8c455ca09bffb24ea64ff6 '']''' 2026-03-20 16:26:25.883046 | controller | - ' logger.go:42: 16:25:44 | watcher-api-scaling/1-deploy-with-defaults | ++ oc 2026-03-20 16:26:25.883053 | controller | get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-20 16:26:25.883057 | controller | - ' logger.go:42: 16:25:44 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-20 16:26:25.883064 | controller | -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.883068 | controller | '']''' 2026-03-20 16:26:25.883072 | controller | - ' logger.go:42: 16:25:44 | watcher-api-scaling/1-deploy-with-defaults | ++ oc 2026-03-20 16:26:25.883076 | controller | get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher' 2026-03-20 16:26:25.883080 | controller | - ' logger.go:42: 16:25:45 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-20 16:26:25.883084 | controller | '''' == '''' '']''' 2026-03-20 16:26:25.883089 | controller | - ' logger.go:42: 16:25:45 | watcher-api-scaling/1-deploy-with-defaults | + exit 2026-03-20 16:26:25.883093 | controller | 0' 2026-03-20 16:26:25.883097 | controller | - ' logger.go:42: 16:25:46 | watcher-api-scaling/1-deploy-with-defaults | running 2026-03-20 16:26:25.883101 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.883105 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.883109 | controller | -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-20 16:26:25.883114 | controller | - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack 2026-03-20 16:26:25.883118 | controller | service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-20 16:26:25.883122 | controller | - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.883126 | controller | == ${SERVICEID} ]' 2026-03-20 16:26:25.883130 | controller | - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-20 16:26:25.883134 | controller | ]' 2026-03-20 16:26:25.883139 | controller | - ' # If we are running the container locally, skip following test' 2026-03-20 16:26:25.883143 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-20 16:26:25.883147 | controller | == "" ]; then' 2026-03-20 16:26:25.883151 | controller | - ' exit 0' 2026-03-20 16:26:25.883157 | controller | - ' fi' 2026-03-20 16:26:25.883163 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-20 16:26:25.883169 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-20 16:26:25.883175 | controller | - ' counter=0' 2026-03-20 16:26:25.883181 | controller | - ' for i in ${env_variables}; do' 2026-03-20 16:26:25.883187 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-20 16:26:25.883193 | controller | - ' echo ${i}' 2026-03-20 16:26:25.883199 | controller | - ' counter=$((counter + 1))' 2026-03-20 16:26:25.883203 | controller | - ' fi' 2026-03-20 16:26:25.883207 | controller | - ' done' 2026-03-20 16:26:25.883212 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-20 16:26:25.883216 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-20 16:26:25.883220 | controller | - ' exit 1' 2026-03-20 16:26:25.883224 | controller | - ' else' 2026-03-20 16:26:25.883231 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-20 16:26:25.883235 | controller | - ' fi' 2026-03-20 16:26:25.883240 | controller | - ' ]' 2026-03-20 16:26:25.883244 | controller | - ' logger.go:42: 16:25:46 | watcher-api-scaling/1-deploy-with-defaults | + oc 2026-03-20 16:26:25.883248 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-20 16:26:25.883253 | controller | -c Name -c Type' 2026-03-20 16:26:25.883259 | controller | - ' logger.go:42: 16:25:46 | watcher-api-scaling/1-deploy-with-defaults | ++ grep 2026-03-20 16:26:25.883265 | controller | -c ''^watcher''' 2026-03-20 16:26:25.883271 | controller | - ' logger.go:42: 16:25:47 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-20 16:26:25.883277 | controller | 1 == 1 '']''' 2026-03-20 16:26:25.883282 | controller | - ' logger.go:42: 16:25:47 | watcher-api-scaling/1-deploy-with-defaults | ++ grep 2026-03-20 16:26:25.883291 | controller | watcher' 2026-03-20 16:26:25.883295 | controller | - ' logger.go:42: 16:25:47 | watcher-api-scaling/1-deploy-with-defaults | ++ oc 2026-03-20 16:26:25.883300 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-20 16:26:25.883304 | controller | -c Name -c Type -c ID' 2026-03-20 16:26:25.883308 | controller | - ' logger.go:42: 16:25:47 | watcher-api-scaling/1-deploy-with-defaults | ++ awk 2026-03-20 16:26:25.883314 | controller | ''{print $1}''' 2026-03-20 16:26:25.883320 | controller | - ' logger.go:42: 16:25:49 | watcher-api-scaling/1-deploy-with-defaults | + SERVICEID=fc1daa96ef8c455ca09bffb24ea64ff6' 2026-03-20 16:26:25.883329 | controller | - ' logger.go:42: 16:25:49 | watcher-api-scaling/1-deploy-with-defaults | ++ oc 2026-03-20 16:26:25.883335 | controller | get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-20 16:26:25.883341 | controller | - ' logger.go:42: 16:25:50 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-20 16:26:25.883347 | controller | fc1daa96ef8c455ca09bffb24ea64ff6 == fc1daa96ef8c455ca09bffb24ea64ff6 '']''' 2026-03-20 16:26:25.883353 | controller | - ' logger.go:42: 16:25:50 | watcher-api-scaling/1-deploy-with-defaults | ++ oc 2026-03-20 16:26:25.883358 | controller | get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-20 16:26:25.883364 | controller | - ' logger.go:42: 16:25:50 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-20 16:26:25.883370 | controller | -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.883376 | controller | '']''' 2026-03-20 16:26:25.883386 | controller | - ' logger.go:42: 16:25:50 | watcher-api-scaling/1-deploy-with-defaults | ++ oc 2026-03-20 16:26:25.883393 | controller | get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher' 2026-03-20 16:26:25.883399 | controller | - ' logger.go:42: 16:25:50 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-20 16:26:25.883404 | controller | '''' == '''' '']''' 2026-03-20 16:26:25.883410 | controller | - ' logger.go:42: 16:25:50 | watcher-api-scaling/1-deploy-with-defaults | + exit 2026-03-20 16:26:25.883416 | controller | 0' 2026-03-20 16:26:25.883422 | controller | - ' logger.go:42: 16:25:51 | watcher-api-scaling/1-deploy-with-defaults | running 2026-03-20 16:26:25.883428 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.883434 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.883440 | controller | -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-20 16:26:25.883446 | controller | - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack 2026-03-20 16:26:25.883451 | controller | service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-20 16:26:25.883457 | controller | - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.883462 | controller | == ${SERVICEID} ]' 2026-03-20 16:26:25.883473 | controller | - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-20 16:26:25.883479 | controller | ]' 2026-03-20 16:26:25.883486 | controller | - ' # If we are running the container locally, skip following test' 2026-03-20 16:26:25.883492 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-20 16:26:25.883498 | controller | == "" ]; then' 2026-03-20 16:26:25.883504 | controller | - ' exit 0' 2026-03-20 16:26:25.883509 | controller | - ' fi' 2026-03-20 16:26:25.883515 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-20 16:26:25.883521 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-20 16:26:25.883527 | controller | - ' counter=0' 2026-03-20 16:26:25.883533 | controller | - ' for i in ${env_variables}; do' 2026-03-20 16:26:25.883539 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-20 16:26:25.883549 | controller | - ' echo ${i}' 2026-03-20 16:26:25.883555 | controller | - ' counter=$((counter + 1))' 2026-03-20 16:26:25.883561 | controller | - ' fi' 2026-03-20 16:26:25.883567 | controller | - ' done' 2026-03-20 16:26:25.883572 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-20 16:26:25.883578 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-20 16:26:25.883584 | controller | - ' exit 1' 2026-03-20 16:26:25.883590 | controller | - ' else' 2026-03-20 16:26:25.883596 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-20 16:26:25.883601 | controller | - ' fi' 2026-03-20 16:26:25.883607 | controller | - ' ]' 2026-03-20 16:26:25.883613 | controller | - ' logger.go:42: 16:25:51 | watcher-api-scaling/1-deploy-with-defaults | + oc 2026-03-20 16:26:25.883619 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-20 16:26:25.883625 | controller | -c Name -c Type' 2026-03-20 16:26:25.883631 | controller | - ' logger.go:42: 16:25:51 | watcher-api-scaling/1-deploy-with-defaults | ++ grep 2026-03-20 16:26:25.883636 | controller | -c ''^watcher''' 2026-03-20 16:26:25.883642 | controller | - ' logger.go:42: 16:25:53 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-20 16:26:25.883648 | controller | 1 == 1 '']''' 2026-03-20 16:26:25.883653 | controller | - ' logger.go:42: 16:25:53 | watcher-api-scaling/1-deploy-with-defaults | ++ grep 2026-03-20 16:26:25.883659 | controller | watcher' 2026-03-20 16:26:25.883664 | controller | - ' logger.go:42: 16:25:53 | watcher-api-scaling/1-deploy-with-defaults | ++ awk 2026-03-20 16:26:25.883668 | controller | ''{print $1}''' 2026-03-20 16:26:25.883673 | controller | - ' logger.go:42: 16:25:53 | watcher-api-scaling/1-deploy-with-defaults | ++ oc 2026-03-20 16:26:25.883677 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-20 16:26:25.883681 | controller | -c Name -c Type -c ID' 2026-03-20 16:26:25.883685 | controller | - ' logger.go:42: 16:25:55 | watcher-api-scaling/1-deploy-with-defaults | + SERVICEID=fc1daa96ef8c455ca09bffb24ea64ff6' 2026-03-20 16:26:25.883689 | controller | - ' logger.go:42: 16:25:55 | watcher-api-scaling/1-deploy-with-defaults | ++ oc 2026-03-20 16:26:25.883711 | controller | get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-20 16:26:25.883716 | controller | - ' logger.go:42: 16:25:55 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-20 16:26:25.883720 | controller | fc1daa96ef8c455ca09bffb24ea64ff6 == fc1daa96ef8c455ca09bffb24ea64ff6 '']''' 2026-03-20 16:26:25.883724 | controller | - ' logger.go:42: 16:25:55 | watcher-api-scaling/1-deploy-with-defaults | ++ oc 2026-03-20 16:26:25.883728 | controller | get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-20 16:26:25.883732 | controller | - ' logger.go:42: 16:25:56 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-20 16:26:25.883737 | controller | -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.883741 | controller | '']''' 2026-03-20 16:26:25.883745 | controller | - ' logger.go:42: 16:25:56 | watcher-api-scaling/1-deploy-with-defaults | ++ oc 2026-03-20 16:26:25.883749 | controller | get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher' 2026-03-20 16:26:25.883753 | controller | - ' logger.go:42: 16:25:56 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-20 16:26:25.883758 | controller | '''' == '''' '']''' 2026-03-20 16:26:25.883770 | controller | - ' logger.go:42: 16:25:56 | watcher-api-scaling/1-deploy-with-defaults | + exit 2026-03-20 16:26:25.883775 | controller | 0' 2026-03-20 16:26:25.883779 | controller | - ' logger.go:42: 16:25:57 | watcher-api-scaling/1-deploy-with-defaults | running 2026-03-20 16:26:25.883783 | controller | command: [sh -c set -euxo pipefail' 2026-03-20 16:26:25.883787 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.883792 | controller | -f value -c Name -c Type |[ $(grep -c ^watcher) == 1 ]' 2026-03-20 16:26:25.883821 | controller | - ' SERVICEID=$(oc exec -n watcher-kuttl-default openstackclient -- openstack 2026-03-20 16:26:25.883825 | controller | service list -f value -c Name -c Type -c ID | grep watcher| awk ''{print $1}'')' 2026-03-20 16:26:25.883830 | controller | - ' [ $(oc get -n watcher-kuttl-default keystoneservice watcher -o jsonpath={.status.serviceID}) 2026-03-20 16:26:25.883834 | controller | == ${SERVICEID} ]' 2026-03-20 16:26:25.883838 | controller | - ' [ -n "$(oc get -n watcher-kuttl-default watcher watcher-kuttl -o jsonpath={.status.hash.dbsync})" 2026-03-20 16:26:25.883842 | controller | ]' 2026-03-20 16:26:25.883847 | controller | - ' # If we are running the container locally, skip following test' 2026-03-20 16:26:25.883851 | controller | - ' if [ "$(oc get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher)" 2026-03-20 16:26:25.883855 | controller | == "" ]; then' 2026-03-20 16:26:25.883859 | controller | - ' exit 0' 2026-03-20 16:26:25.883863 | controller | - ' fi' 2026-03-20 16:26:25.883867 | controller | - ' env_variables=$(oc set env $(oc get pods -n openstack-operators -o name 2026-03-20 16:26:25.883872 | controller | -l openstack.org/operator-name=watcher) -n openstack-operators --list)' 2026-03-20 16:26:25.883876 | controller | - ' counter=0' 2026-03-20 16:26:25.883880 | controller | - ' for i in ${env_variables}; do' 2026-03-20 16:26:25.883884 | controller | - ' if echo ${i} | grep ''_URL_DEFAULT'' &> /dev/null; then' 2026-03-20 16:26:25.883888 | controller | - ' echo ${i}' 2026-03-20 16:26:25.883892 | controller | - ' counter=$((counter + 1))' 2026-03-20 16:26:25.883897 | controller | - ' fi' 2026-03-20 16:26:25.883901 | controller | - ' done' 2026-03-20 16:26:25.883905 | controller | - ' if [ ${counter} -lt 3 ]; then' 2026-03-20 16:26:25.883909 | controller | - ' echo "Error: Less than 3 _URL_DEFAULT variables found."' 2026-03-20 16:26:25.883913 | controller | - ' exit 1' 2026-03-20 16:26:25.883917 | controller | - ' else' 2026-03-20 16:26:25.883922 | controller | - ' echo "Success: ${counter} _URL_DEFAULT variables found."' 2026-03-20 16:26:25.883926 | controller | - ' fi' 2026-03-20 16:26:25.883930 | controller | - ' ]' 2026-03-20 16:26:25.883934 | controller | - ' logger.go:42: 16:25:57 | watcher-api-scaling/1-deploy-with-defaults | + oc 2026-03-20 16:26:25.883938 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-20 16:26:25.883942 | controller | -c Name -c Type' 2026-03-20 16:26:25.883947 | controller | - ' logger.go:42: 16:25:57 | watcher-api-scaling/1-deploy-with-defaults | ++ grep 2026-03-20 16:26:25.883951 | controller | -c ''^watcher''' 2026-03-20 16:26:25.883955 | controller | - ' logger.go:42: 16:25:59 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-20 16:26:25.883959 | controller | 1 == 1 '']''' 2026-03-20 16:26:25.883963 | controller | - ' logger.go:42: 16:25:59 | watcher-api-scaling/1-deploy-with-defaults | ++ oc 2026-03-20 16:26:25.883969 | controller | exec -n watcher-kuttl-default openstackclient -- openstack service list -f value 2026-03-20 16:26:25.883975 | controller | -c Name -c Type -c ID' 2026-03-20 16:26:25.883981 | controller | - ' logger.go:42: 16:25:59 | watcher-api-scaling/1-deploy-with-defaults | ++ awk 2026-03-20 16:26:25.883987 | controller | ''{print $1}''' 2026-03-20 16:26:25.883993 | controller | - ' logger.go:42: 16:25:59 | watcher-api-scaling/1-deploy-with-defaults | ++ grep 2026-03-20 16:26:25.883998 | controller | watcher' 2026-03-20 16:26:25.884004 | controller | - ' logger.go:42: 16:26:01 | watcher-api-scaling/1-deploy-with-defaults | + SERVICEID=fc1daa96ef8c455ca09bffb24ea64ff6' 2026-03-20 16:26:25.884010 | controller | - ' logger.go:42: 16:26:01 | watcher-api-scaling/1-deploy-with-defaults | ++ oc 2026-03-20 16:26:25.884016 | controller | get -n watcher-kuttl-default keystoneservice watcher -o ''jsonpath={.status.serviceID}''' 2026-03-20 16:26:25.884022 | controller | - ' logger.go:42: 16:26:01 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-20 16:26:25.884028 | controller | fc1daa96ef8c455ca09bffb24ea64ff6 == fc1daa96ef8c455ca09bffb24ea64ff6 '']''' 2026-03-20 16:26:25.884036 | controller | - ' logger.go:42: 16:26:01 | watcher-api-scaling/1-deploy-with-defaults | ++ oc 2026-03-20 16:26:25.884040 | controller | get -n watcher-kuttl-default watcher watcher-kuttl -o ''jsonpath={.status.hash.dbsync}''' 2026-03-20 16:26:25.884045 | controller | - ' logger.go:42: 16:26:01 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-20 16:26:25.884049 | controller | -n n8dh549h5hbfh56fh86h5f9hc8h76h694hcdh589h655h666hf7h5ch94h588h64fhb7h684h576h54h5f7h64h5d8hd5h5c8h654h554h658h647q 2026-03-20 16:26:25.884053 | controller | '']''' 2026-03-20 16:26:25.884057 | controller | - ' logger.go:42: 16:26:01 | watcher-api-scaling/1-deploy-with-defaults | ++ oc 2026-03-20 16:26:25.884062 | controller | get pods -n openstack-operators -o name -l openstack.org/operator-name=watcher' 2026-03-20 16:26:25.884066 | controller | - ' logger.go:42: 16:26:02 | watcher-api-scaling/1-deploy-with-defaults | + ''['' 2026-03-20 16:26:25.884071 | controller | '''' == '''' '']''' 2026-03-20 16:26:25.884077 | controller | - ' logger.go:42: 16:26:02 | watcher-api-scaling/1-deploy-with-defaults | + exit 2026-03-20 16:26:25.884082 | controller | 0' 2026-03-20 16:26:25.884088 | controller | - ' logger.go:42: 16:26:02 | watcher-api-scaling/1-deploy-with-defaults | test 2026-03-20 16:26:25.884094 | controller | step completed 1-deploy-with-defaults' 2026-03-20 16:26:25.884100 | controller | - ' logger.go:42: 16:26:02 | watcher-api-scaling/2-scale-up-watcher-api | starting 2026-03-20 16:26:25.884106 | controller | test step 2-scale-up-watcher-api' 2026-03-20 16:26:25.884116 | controller | - ' logger.go:42: 16:26:02 | watcher-api-scaling/2-scale-up-watcher-api | running 2026-03-20 16:26:25.884125 | controller | command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type=''json'' -p=''[{"op": 2026-03-20 16:26:25.884130 | controller | "replace", "path": "/spec/apiServiceTemplate/replicas", "value":3}]''' 2026-03-20 16:26:25.884135 | controller | - ' ]' 2026-03-20 16:26:25.884139 | controller | - ' logger.go:42: 16:26:02 | watcher-api-scaling/2-scale-up-watcher-api | watcher.watcher.openstack.org/watcher-kuttl 2026-03-20 16:26:25.884143 | controller | patched' 2026-03-20 16:26:25.884148 | controller | - ' logger.go:42: 16:26:13 | watcher-api-scaling/2-scale-up-watcher-api | test 2026-03-20 16:26:25.884152 | controller | step completed 2-scale-up-watcher-api' 2026-03-20 16:26:25.884156 | controller | - ' logger.go:42: 16:26:13 | watcher-api-scaling/3-scale-down-watcher-api | starting 2026-03-20 16:26:25.884160 | controller | test step 3-scale-down-watcher-api' 2026-03-20 16:26:25.884164 | controller | - ' logger.go:42: 16:26:13 | watcher-api-scaling/3-scale-down-watcher-api | running 2026-03-20 16:26:25.884168 | controller | command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type=''json'' -p=''[{"op": 2026-03-20 16:26:25.884172 | controller | "replace", "path": "/spec/apiServiceTemplate/replicas", "value":1}]''' 2026-03-20 16:26:25.884177 | controller | - ' ]' 2026-03-20 16:26:25.884181 | controller | - ' logger.go:42: 16:26:13 | watcher-api-scaling/3-scale-down-watcher-api | watcher.watcher.openstack.org/watcher-kuttl 2026-03-20 16:26:25.884185 | controller | patched' 2026-03-20 16:26:25.884189 | controller | - ' logger.go:42: 16:26:16 | watcher-api-scaling/3-scale-down-watcher-api | test 2026-03-20 16:26:25.884193 | controller | step completed 3-scale-down-watcher-api' 2026-03-20 16:26:25.884198 | controller | - ' logger.go:42: 16:26:16 | watcher-api-scaling/4-scale-down-zero-watcher-api 2026-03-20 16:26:25.884202 | controller | | starting test step 4-scale-down-zero-watcher-api' 2026-03-20 16:26:25.884206 | controller | - ' logger.go:42: 16:26:16 | watcher-api-scaling/4-scale-down-zero-watcher-api 2026-03-20 16:26:25.884212 | controller | | running command: [sh -c oc patch watcher -n $NAMESPACE watcher-kuttl --type=''json'' 2026-03-20 16:26:25.884217 | controller | -p=''[{"op": "replace", "path": "/spec/apiServiceTemplate/replicas", "value":0}]''' 2026-03-20 16:26:25.884221 | controller | - ' ]' 2026-03-20 16:26:25.884225 | controller | - ' logger.go:42: 16:26:17 | watcher-api-scaling/4-scale-down-zero-watcher-api 2026-03-20 16:26:25.884229 | controller | | watcher.watcher.openstack.org/watcher-kuttl patched' 2026-03-20 16:26:25.884233 | controller | - ' logger.go:42: 16:26:18 | watcher-api-scaling/4-scale-down-zero-watcher-api 2026-03-20 16:26:25.884241 | controller | | test step completed 4-scale-down-zero-watcher-api' 2026-03-20 16:26:25.884245 | controller | - ' logger.go:42: 16:26:18 | watcher-api-scaling/5-cleanup-watcher | starting test 2026-03-20 16:26:25.884249 | controller | step 5-cleanup-watcher' 2026-03-20 16:26:25.884253 | controller | - ' logger.go:42: 16:26:18 | watcher-api-scaling/5-cleanup-watcher | running command: 2026-03-20 16:26:25.884258 | controller | [sh -c set -ex' 2026-03-20 16:26:25.884262 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.884266 | controller | -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]' 2026-03-20 16:26:25.884270 | controller | - ' ]' 2026-03-20 16:26:25.884274 | controller | - ' logger.go:42: 16:26:18 | watcher-api-scaling/5-cleanup-watcher | + oc exec 2026-03-20 16:26:25.884279 | controller | -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-20 16:26:25.884283 | controller | -c Type' 2026-03-20 16:26:25.884287 | controller | - ' logger.go:42: 16:26:18 | watcher-api-scaling/5-cleanup-watcher | ++ grep -c 2026-03-20 16:26:25.884291 | controller | ''^watcher''' 2026-03-20 16:26:25.884295 | controller | - ' logger.go:42: 16:26:20 | watcher-api-scaling/5-cleanup-watcher | + ''['' 1 2026-03-20 16:26:25.884299 | controller | == 0 '']''' 2026-03-20 16:26:25.884304 | controller | - ' logger.go:42: 16:26:22 | watcher-api-scaling/5-cleanup-watcher | running command: 2026-03-20 16:26:25.884308 | controller | [sh -c set -ex' 2026-03-20 16:26:25.884312 | controller | - ' oc exec -n watcher-kuttl-default openstackclient -- openstack service list 2026-03-20 16:26:25.884316 | controller | -f value -c Name -c Type | [ $(grep -c ^watcher) == 0 ]' 2026-03-20 16:26:25.884320 | controller | - ' ]' 2026-03-20 16:26:25.884324 | controller | - ' logger.go:42: 16:26:22 | watcher-api-scaling/5-cleanup-watcher | + oc exec 2026-03-20 16:26:25.884328 | controller | -n watcher-kuttl-default openstackclient -- openstack service list -f value -c Name 2026-03-20 16:26:25.884333 | controller | -c Type' 2026-03-20 16:26:25.884337 | controller | - ' logger.go:42: 16:26:22 | watcher-api-scaling/5-cleanup-watcher | ++ grep -c 2026-03-20 16:26:25.884341 | controller | ''^watcher''' 2026-03-20 16:26:25.884345 | controller | - ' logger.go:42: 16:26:24 | watcher-api-scaling/5-cleanup-watcher | + ''['' 0 2026-03-20 16:26:25.884350 | controller | == 0 '']''' 2026-03-20 16:26:25.884354 | controller | - ' logger.go:42: 16:26:24 | watcher-api-scaling/5-cleanup-watcher | test step 2026-03-20 16:26:25.884358 | controller | completed 5-cleanup-watcher' 2026-03-20 16:26:25.884362 | controller | - ' logger.go:42: 16:26:24 | watcher-api-scaling | skipping kubernetes event logging' 2026-03-20 16:26:25.884366 | controller | - === CONT kuttl/harness/deps 2026-03-20 16:26:25.884370 | controller | - ' logger.go:42: 16:26:24 | deps | Ignoring infra.yaml as it does not match file 2026-03-20 16:26:25.884375 | controller | name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$' 2026-03-20 16:26:25.884379 | controller | - ' logger.go:42: 16:26:24 | deps | Ignoring keystone.yaml as it does not match 2026-03-20 16:26:25.884383 | controller | file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$' 2026-03-20 16:26:25.884387 | controller | - ' logger.go:42: 16:26:24 | deps | Ignoring kustomization.yaml as it does not 2026-03-20 16:26:25.884394 | controller | match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$' 2026-03-20 16:26:25.972422 | controller | - ' logger.go:42: 16:26:24 | deps | Ignoring namespace.yaml as it does not match 2026-03-20 16:26:25.972482 | controller | file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$' 2026-03-20 16:26:25.972488 | controller | - ' logger.go:42: 16:26:24 | deps | Ignoring telemetry.yaml as it does not match 2026-03-20 16:26:25.972493 | controller | file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$' 2026-03-20 16:26:25.972497 | controller | - ' logger.go:42: 16:26:24 | deps | Skipping creation of user-supplied namespace: 2026-03-20 16:26:25.972502 | controller | watcher-kuttl-default' 2026-03-20 16:26:25.972508 | controller | - ' logger.go:42: 16:26:24 | deps | skipping kubernetes event logging' 2026-03-20 16:26:25.972518 | controller | - === CONT kuttl/harness/common 2026-03-20 16:26:25.972523 | controller | - ' logger.go:42: 16:26:24 | common | Ignoring cleanup-assert.yaml as it does not 2026-03-20 16:26:25.972527 | controller | match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$' 2026-03-20 16:26:25.972531 | controller | - ' logger.go:42: 16:26:24 | common | Ignoring cleanup-errors.yaml as it does not 2026-03-20 16:26:25.972535 | controller | match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$' 2026-03-20 16:26:25.972539 | controller | - ' logger.go:42: 16:26:24 | common | Ignoring cleanup-watcher.yaml as it does 2026-03-20 16:26:25.972544 | controller | not match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$' 2026-03-20 16:26:25.972548 | controller | - ' logger.go:42: 16:26:24 | common | Ignoring deploy-with-defaults.yaml as it 2026-03-20 16:26:25.972552 | controller | does not match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$' 2026-03-20 16:26:25.972556 | controller | - ' logger.go:42: 16:26:24 | common | Skipping creation of user-supplied namespace: 2026-03-20 16:26:25.972560 | controller | watcher-kuttl-default' 2026-03-20 16:26:25.972565 | controller | - ' logger.go:42: 16:26:24 | common | skipping kubernetes event logging' 2026-03-20 16:26:25.972569 | controller | - === NAME kuttl 2026-03-20 16:26:25.972573 | controller | - ' harness.go:406: run tests finished' 2026-03-20 16:26:25.972578 | controller | - ' harness.go:514: cleaning up' 2026-03-20 16:26:25.972582 | controller | - ' harness.go:571: removing temp folder: ""' 2026-03-20 16:26:25.972590 | controller | - '--- FAIL: kuttl (1493.95s)' 2026-03-20 16:26:25.972595 | controller | - ' --- FAIL: kuttl/harness (0.00s)' 2026-03-20 16:26:25.972599 | controller | - ' --- FAIL: kuttl/harness/appcred-tests (962.71s)' 2026-03-20 16:26:25.972603 | controller | - ' --- PASS: kuttl/harness/watcher-notification (37.61s)' 2026-03-20 16:26:25.972608 | controller | - ' --- PASS: kuttl/harness/watcher-topology (31.57s)' 2026-03-20 16:26:25.972612 | controller | - ' --- PASS: kuttl/harness/watcher-tls-certs-change (36.81s)' 2026-03-20 16:26:25.972616 | controller | - ' --- PASS: kuttl/harness/watcher-tls (127.41s)' 2026-03-20 16:26:25.972621 | controller | - ' --- PASS: kuttl/harness/watcher-rmquser (34.91s)' 2026-03-20 16:26:25.972625 | controller | - ' --- PASS: kuttl/harness/watcher (79.13s)' 2026-03-20 16:26:25.972629 | controller | - ' --- PASS: kuttl/harness/watcher-cinder (126.78s)' 2026-03-20 16:26:25.972633 | controller | - ' --- PASS: kuttl/harness/watcher-api-scaling (56.98s)' 2026-03-20 16:26:25.972637 | controller | - ' --- PASS: kuttl/harness/deps (0.00s)' 2026-03-20 16:26:25.972641 | controller | - ' --- PASS: kuttl/harness/common (0.00s)' 2026-03-20 16:26:25.972646 | controller | - FAIL 2026-03-20 16:26:25.972658 | controller | 2026-03-20 16:26:26.037270 | controller | NO MORE HOSTS LEFT ************************************************************* 2026-03-20 16:26:26.037383 | controller | 2026-03-20 16:26:26.037391 | controller | PLAY RECAP ********************************************************************* 2026-03-20 16:26:26.037396 | controller | controller : ok=2 changed=0 unreachable=0 failed=1 skipped=1 rescued=0 ignored=0 2026-03-20 16:26:26.037401 | controller | 2026-03-20 16:26:26.037405 | controller | Friday 20 March 2026 16:26:25 +0000 (0:24:55.644) 0:24:56.041 ********** 2026-03-20 16:26:26.037409 | controller | =============================================================================== 2026-03-20 16:26:26.037413 | controller | run kuttl test suite from operator Makefile -------------------------- 1495.64s 2026-03-20 16:26:26.037419 | controller | run_hook : Loop on hooks for pre_kuttl_from_operator -------------------- 0.14s 2026-03-20 16:26:26.037425 | controller | run_hook : Assert single hooks are all mappings ------------------------- 0.10s 2026-03-20 16:26:26.037431 | controller | run_hook : Assert parameters are valid ---------------------------------- 0.07s 2026-03-20 16:26:26.037437 | controller | Run hooks before running kuttl tests ------------------------------------ 0.04s 2026-03-20 16:26:26.037443 | controller | Friday 20 March 2026 16:26:25 +0000 (0:24:55.644) 0:24:56.040 ********** 2026-03-20 16:26:26.037449 | controller | =============================================================================== 2026-03-20 16:26:26.037462 | controller | ansible.builtin.command ---------------------------------------------- 1495.64s 2026-03-20 16:26:26.037466 | controller | run_hook ---------------------------------------------------------------- 0.31s 2026-03-20 16:26:26.037470 | controller | ansible.builtin.include_role -------------------------------------------- 0.04s 2026-03-20 16:26:26.037475 | controller | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2026-03-20 16:26:26.037479 | controller | total ---------------------------------------------------------------- 1496.00s 2026-03-20 16:26:27.774243 | controller | ERROR 2026-03-20 16:26:27.774490 | controller | { 2026-03-20 16:26:27.774531 | controller | "delta": "0:24:56.606788", 2026-03-20 16:26:27.774559 | controller | "end": "2026-03-20 16:26:26.042368", 2026-03-20 16:26:27.774584 | controller | "msg": "non-zero return code", 2026-03-20 16:26:27.774607 | controller | "rc": 2, 2026-03-20 16:26:27.774630 | controller | "start": "2026-03-20 16:01:29.435580" 2026-03-20 16:26:27.774653 | controller | } failure 2026-03-20 16:26:28.391816 | 2026-03-20 16:26:28.391954 | PLAY RECAP 2026-03-20 16:26:28.391998 | controller | ok: 0 changed: 0 unreachable: 0 failed: 1 skipped: 0 rescued: 0 ignored: 0 2026-03-20 16:26:28.392019 | 2026-03-20 16:26:28.582558 | RUN END RESULT_NORMAL: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/kuttl/kuttl-from-operator-run.yaml@main] 2026-03-20 16:26:28.591346 | POST-RUN START: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/e2e-collect-logs.yml@main] 2026-03-20 16:26:29.284590 | 2026-03-20 16:26:29.284771 | PLAY [Run ci/playbooks/e2e-collect-logs.yml] 2026-03-20 16:26:29.329996 | 2026-03-20 16:26:29.330090 | TASK [Filter out host if needed] 2026-03-20 16:26:29.341498 | controller | skipping: end_host conditional evaluated to False, continuing execution for controller 2026-03-20 16:26:29.347451 | 2026-03-20 16:26:29.347512 | TASK [Filter out host if needed] 2026-03-20 16:26:29.362152 | 2026-03-20 16:26:29.362225 | TASK [Ensure we have the ci-framework on host] 2026-03-20 16:26:30.927859 | controller | ok 2026-03-20 16:26:30.941138 | 2026-03-20 16:26:30.941270 | TASK [End host if no basedir] 2026-03-20 16:26:30.955405 | controller | skipping: end_host conditional evaluated to False, continuing execution for controller 2026-03-20 16:26:30.967157 | 2026-03-20 16:26:30.967273 | TASK [Read base centos-9 scenarios] 2026-03-20 16:26:30.990049 | controller | ok 2026-03-20 16:26:31.082210 | 2026-03-20 16:26:31.082289 | TASK [cifmw_helpers : Fail if file is not yaml or yml extension] 2026-03-20 16:26:31.136604 | controller | skipping: Conditional result was False 2026-03-20 16:26:31.179579 | 2026-03-20 16:26:31.179747 | TASK [cifmw_helpers : Check if file is available] 2026-03-20 16:26:32.216780 | controller | ok 2026-03-20 16:26:32.229964 | 2026-03-20 16:26:32.230104 | TASK [cifmw_helpers : Create temporary directory] 2026-03-20 16:26:32.267517 | controller | skipping: Conditional result was False 2026-03-20 16:26:32.282617 | 2026-03-20 16:26:32.282785 | TASK [cifmw_helpers : Read vars] 2026-03-20 16:26:32.326886 | 2026-03-20 16:26:32.327164 | TASK [cifmw_helpers : Create new variable files with content] 2026-03-20 16:26:32.374054 | 2026-03-20 16:26:32.374251 | TASK [cifmw_helpers : Include vars] 2026-03-20 16:26:32.407822 | 2026-03-20 16:26:32.408028 | TASK [cifmw_helpers : Remove temporary directory] 2026-03-20 16:26:32.442432 | controller | skipping: Conditional result was False 2026-03-20 16:26:32.459718 | 2026-03-20 16:26:32.459812 | TASK [run_hook : Assert parameters are valid] 2026-03-20 16:26:32.533152 | controller | ok: All assertions passed 2026-03-20 16:26:32.538861 | 2026-03-20 16:26:32.538923 | TASK [run_hook : Assert single hooks are all mappings] 2026-03-20 16:26:32.609157 | controller | ok: All assertions passed 2026-03-20 16:26:32.615940 | 2026-03-20 16:26:32.616003 | LOOP [run_hook : Loop on hooks for pre_logs] 2026-03-20 16:26:32.692999 | 2026-03-20 16:26:32.693155 | TASK [cifmw_setup : Ensure cifmw_basedir param is set] 2026-03-20 16:26:32.706904 | controller | skipping: Conditional result was False 2026-03-20 16:26:32.714595 | 2026-03-20 16:26:32.714685 | TASK [Read artifacts parameters dir and set as facts] 2026-03-20 16:26:32.734023 | controller | ok 2026-03-20 16:26:32.753153 | 2026-03-20 16:26:32.753229 | TASK [cifmw_helpers : Check directory is available] 2026-03-20 16:26:33.016095 | controller | ok 2026-03-20 16:26:33.023061 | 2026-03-20 16:26:33.023131 | TASK [cifmw_helpers : Find yaml files] 2026-03-20 16:26:33.367822 | controller | ok: All paths examined 2026-03-20 16:26:33.375182 | 2026-03-20 16:26:33.375242 | TASK [cifmw_helpers : Print available yaml files] 2026-03-20 16:26:33.404579 | controller | ok: Found yaml files to parse: ['/home/zuul/ci-framework-data/artifacts/parameters/zuul-params.yml', '/home/zuul/ci-framework-data/artifacts/parameters/install-yamls-params.yml', '/home/zuul/ci-framework-data/artifacts/parameters/custom-params.yml', '/home/zuul/ci-framework-data/artifacts/parameters/openshift-login-params.yml'] 2026-03-20 16:26:33.411266 | 2026-03-20 16:26:33.411327 | LOOP [cifmw_helpers : Create files on localhost and use include_vars] 2026-03-20 16:26:33.435800 | controller | Output suppressed because no_log was given 2026-03-20 16:26:33.436717 | 2026-03-20 16:26:33.445777 | controller | Output suppressed because no_log was given 2026-03-20 16:26:33.453436 | controller | Output suppressed because no_log was given 2026-03-20 16:26:33.459191 | controller | Output suppressed because no_log was given 2026-03-20 16:26:33.479722 | controller | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/untrusted/project_0/github.com/openstack-k8s-operators/ci-framework/roles/cifmw_helpers/tasks/include_file.yml 2026-03-20 16:26:33.484561 | controller | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/untrusted/project_0/github.com/openstack-k8s-operators/ci-framework/roles/cifmw_helpers/tasks/include_file.yml 2026-03-20 16:26:33.489141 | controller | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/untrusted/project_0/github.com/openstack-k8s-operators/ci-framework/roles/cifmw_helpers/tasks/include_file.yml 2026-03-20 16:26:33.493618 | controller | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/untrusted/project_0/github.com/openstack-k8s-operators/ci-framework/roles/cifmw_helpers/tasks/include_file.yml 2026-03-20 16:26:33.501490 | 2026-03-20 16:26:33.501550 | TASK [cifmw_helpers : Fail if file is not yaml or yml extension] 2026-03-20 16:26:33.526018 | controller | skipping: Conditional result was False 2026-03-20 16:26:33.532709 | 2026-03-20 16:26:33.532772 | TASK [cifmw_helpers : Check if file is available] 2026-03-20 16:26:33.774522 | controller | ok 2026-03-20 16:26:33.780762 | 2026-03-20 16:26:33.780820 | TASK [cifmw_helpers : Create temporary directory] 2026-03-20 16:26:34.201990 | controller -> localhost | changed 2026-03-20 16:26:34.216706 | 2026-03-20 16:26:34.216863 | TASK [cifmw_helpers : Read vars] 2026-03-20 16:26:34.591895 | controller | Output suppressed because no_log was given 2026-03-20 16:26:34.604952 | 2026-03-20 16:26:34.605080 | TASK [cifmw_helpers : Create new variable files with content] 2026-03-20 16:26:35.296396 | controller -> localhost | Output suppressed because no_log was given 2026-03-20 16:26:35.310560 | 2026-03-20 16:26:35.310724 | TASK [cifmw_helpers : Include vars] 2026-03-20 16:26:35.379091 | controller -> localhost | Output suppressed because no_log was given 2026-03-20 16:26:35.393406 | 2026-03-20 16:26:35.393533 | TASK [cifmw_helpers : Remove temporary directory] 2026-03-20 16:26:35.858371 | controller -> localhost | changed 2026-03-20 16:26:35.873796 | 2026-03-20 16:26:35.873928 | TASK [cifmw_helpers : Fail if file is not yaml or yml extension] 2026-03-20 16:26:35.911781 | controller | skipping: Conditional result was False 2026-03-20 16:26:35.928013 | 2026-03-20 16:26:35.928150 | TASK [cifmw_helpers : Check if file is available] 2026-03-20 16:26:36.191018 | controller | ok 2026-03-20 16:26:36.205108 | 2026-03-20 16:26:36.205236 | TASK [cifmw_helpers : Create temporary directory] 2026-03-20 16:26:36.497814 | controller -> localhost | changed 2026-03-20 16:26:36.513084 | 2026-03-20 16:26:36.513216 | TASK [cifmw_helpers : Read vars] 2026-03-20 16:26:36.727181 | controller | Output suppressed because no_log was given 2026-03-20 16:26:36.739773 | 2026-03-20 16:26:36.739900 | TASK [cifmw_helpers : Create new variable files with content] 2026-03-20 16:26:37.224462 | controller -> localhost | Output suppressed because no_log was given 2026-03-20 16:26:37.239087 | 2026-03-20 16:26:37.239223 | TASK [cifmw_helpers : Include vars] 2026-03-20 16:26:37.328704 | controller -> localhost | Output suppressed because no_log was given 2026-03-20 16:26:37.337914 | 2026-03-20 16:26:37.338007 | TASK [cifmw_helpers : Remove temporary directory] 2026-03-20 16:26:37.693299 | controller -> localhost | changed 2026-03-20 16:26:37.701901 | 2026-03-20 16:26:37.701985 | TASK [cifmw_helpers : Fail if file is not yaml or yml extension] 2026-03-20 16:26:37.737702 | controller | skipping: Conditional result was False 2026-03-20 16:26:37.752750 | 2026-03-20 16:26:37.753019 | TASK [cifmw_helpers : Check if file is available] 2026-03-20 16:26:37.996539 | controller | ok 2026-03-20 16:26:38.011496 | 2026-03-20 16:26:38.011634 | TASK [cifmw_helpers : Create temporary directory] 2026-03-20 16:26:38.325302 | controller -> localhost | changed 2026-03-20 16:26:38.334389 | 2026-03-20 16:26:38.334454 | TASK [cifmw_helpers : Read vars] 2026-03-20 16:26:38.558108 | controller | Output suppressed because no_log was given 2026-03-20 16:26:38.571695 | 2026-03-20 16:26:38.571831 | TASK [cifmw_helpers : Create new variable files with content] 2026-03-20 16:26:39.066806 | controller -> localhost | Output suppressed because no_log was given 2026-03-20 16:26:39.077136 | 2026-03-20 16:26:39.077226 | TASK [cifmw_helpers : Include vars] 2026-03-20 16:26:39.134482 | controller -> localhost | Output suppressed because no_log was given 2026-03-20 16:26:39.151789 | 2026-03-20 16:26:39.151934 | TASK [cifmw_helpers : Remove temporary directory] 2026-03-20 16:26:39.453854 | controller -> localhost | changed 2026-03-20 16:26:39.461702 | 2026-03-20 16:26:39.461772 | TASK [cifmw_helpers : Fail if file is not yaml or yml extension] 2026-03-20 16:26:39.496345 | controller | skipping: Conditional result was False 2026-03-20 16:26:39.504225 | 2026-03-20 16:26:39.504294 | TASK [cifmw_helpers : Check if file is available] 2026-03-20 16:26:39.790792 | controller | ok 2026-03-20 16:26:39.799715 | 2026-03-20 16:26:39.799800 | TASK [cifmw_helpers : Create temporary directory] 2026-03-20 16:26:40.082281 | controller -> localhost | changed 2026-03-20 16:26:40.099781 | 2026-03-20 16:26:40.099940 | TASK [cifmw_helpers : Read vars] 2026-03-20 16:26:40.388521 | controller | Output suppressed because no_log was given 2026-03-20 16:26:40.402387 | 2026-03-20 16:26:40.402525 | TASK [cifmw_helpers : Create new variable files with content] 2026-03-20 16:26:40.900407 | controller -> localhost | Output suppressed because no_log was given 2026-03-20 16:26:40.915377 | 2026-03-20 16:26:40.915509 | TASK [cifmw_helpers : Include vars] 2026-03-20 16:26:40.973766 | controller -> localhost | Output suppressed because no_log was given 2026-03-20 16:26:40.990511 | 2026-03-20 16:26:40.990640 | TASK [cifmw_helpers : Remove temporary directory] 2026-03-20 16:26:41.244303 | controller -> localhost | changed 2026-03-20 16:26:41.272959 | 2026-03-20 16:26:41.273097 | TASK [cifmw_setup : Set custom cifmw PATH reusable fact] 2026-03-20 16:26:41.301255 | controller | skipping: Conditional result was False 2026-03-20 16:26:41.316612 | 2026-03-20 16:26:41.316778 | TASK [cifmw_setup : Set destination folder for the logs] 2026-03-20 16:26:41.367276 | controller | ok 2026-03-20 16:26:41.383318 | 2026-03-20 16:26:41.383454 | TASK [artifacts : Verify if "success" flag exists after successful tests execution] 2026-03-20 16:26:41.637932 | controller | ok 2026-03-20 16:26:41.651497 | 2026-03-20 16:26:41.651634 | TASK [artifacts : Ensure we have at least minimal ansible facts] 2026-03-20 16:26:42.298577 | controller | ok 2026-03-20 16:26:42.314892 | 2026-03-20 16:26:42.315031 | LOOP [artifacts : Ensure base directory exists] 2026-03-20 16:26:42.545691 | controller | ok: "artifacts" 2026-03-20 16:26:42.735828 | controller | ok: "logs" 2026-03-20 16:26:42.757252 | 2026-03-20 16:26:42.757404 | TASK [artifacts : Load generated hook environment for further usage] 2026-03-20 16:26:42.809062 | controller | ERROR 2026-03-20 16:26:42.809437 | controller | { 2026-03-20 16:26:42.809501 | controller | "ansible_facts": {}, 2026-03-20 16:26:42.809543 | controller | "ansible_included_var_files": [], 2026-03-20 16:26:42.809580 | controller | "message": "/home/zuul/ci-framework-data/artifacts directory does not exist" 2026-03-20 16:26:42.809616 | controller | } 2026-03-20 16:26:42.809698 | controller | ERROR: Ignoring Errors 2026-03-20 16:26:42.823996 | 2026-03-20 16:26:42.824158 | TASK [artifacts : Gather parameter passed to the playbook] 2026-03-20 16:26:43.661008 | controller | changed 2026-03-20 16:26:43.674654 | 2026-03-20 16:26:43.674839 | TASK [artifacts : Gather facts] 2026-03-20 16:26:44.310655 | controller | changed 2026-03-20 16:26:44.327480 | 2026-03-20 16:26:44.327776 | TASK [artifacts : Copy some network configuration files] 2026-03-20 16:26:44.924651 | controller | changed 2026-03-20 16:26:44.938289 | 2026-03-20 16:26:44.938418 | TASK [artifacts : Get installed packages list] 2026-03-20 16:26:45.910194 | controller | ok 2026-03-20 16:26:45.975888 | 2026-03-20 16:26:45.976012 | TASK [artifacts : Output package list] 2026-03-20 16:26:46.677352 | controller | changed 2026-03-20 16:26:46.694539 | 2026-03-20 16:26:46.694641 | LOOP [artifacts : Cleanup unnecesary directories] 2026-03-20 16:26:47.009087 | controller | changed: "/home/zuul/ci-framework-data/artifacts/manifests/operator" 2026-03-20 16:26:47.009728 | controller | ok: All items complete 2026-03-20 16:26:47.009805 | 2026-03-20 16:26:47.069748 | 2026-03-20 16:26:47.069972 | TASK [os_must_gather : Ensure directories are present] 2026-03-20 16:26:47.319499 | controller | changed 2026-03-20 16:26:47.345992 | 2026-03-20 16:26:47.346084 | LOOP [os_must_gather : Construct project change list] 2026-03-20 16:26:47.443204 | controller | ok: 2026-03-20 16:26:47.443437 | controller | { 2026-03-20 16:26:47.443471 | controller | "branch": "main", 2026-03-20 16:26:47.443496 | controller | "change": "354", 2026-03-20 16:26:47.443521 | controller | "change_message": "Disable horizon deployment in master release EDPM jobs\n\nHorizon packages in RDO centos10-master is broken and fails to start. Given that we are not running automated horizon tests in the edpm deployment jobs, let's disable it in the jobs deploying from master.\r\n\r\nAssisted-By: Cursor", 2026-03-20 16:26:47.443545 | controller | "change_url": "https://github.com/openstack-k8s-operators/watcher-operator/pull/354", 2026-03-20 16:26:47.443569 | controller | "commit_id": "ee00c2d330b27d46c48ac29a20680b56ca50df3c", 2026-03-20 16:26:47.443591 | controller | "patchset": "ee00c2d330b27d46c48ac29a20680b56ca50df3c", 2026-03-20 16:26:47.443613 | controller | "project": { 2026-03-20 16:26:47.443635 | controller | "canonical_hostname": "github.com", 2026-03-20 16:26:47.443656 | controller | "canonical_name": "github.com/openstack-k8s-operators/watcher-operator", 2026-03-20 16:26:47.443704 | controller | "name": "openstack-k8s-operators/watcher-operator", 2026-03-20 16:26:47.443727 | controller | "short_name": "watcher-operator", 2026-03-20 16:26:47.443748 | controller | "src_dir": "src/github.com/openstack-k8s-operators/watcher-operator" 2026-03-20 16:26:47.443770 | controller | }, 2026-03-20 16:26:47.443791 | controller | "topic": null 2026-03-20 16:26:47.443812 | controller | } 2026-03-20 16:26:47.537059 | 2026-03-20 16:26:47.537203 | TASK [os_must_gather : Get git tag for image tagging] 2026-03-20 16:26:48.151655 | controller | skipping: Conditional result was False 2026-03-20 16:26:48.194135 | 2026-03-20 16:26:48.194301 | TASK [os_must_gather : Debug git show-ref] 2026-03-20 16:26:48.232011 | controller | skipping: Conditional result was False 2026-03-20 16:26:48.277461 | 2026-03-20 16:26:48.277563 | TASK [os_must_gather : Build openstack-must-gather container] 2026-03-20 16:26:48.313196 | controller | skipping: Conditional result was False 2026-03-20 16:26:48.340582 | 2026-03-20 16:26:48.340710 | TASK [os_must_gather : Push openstack-must-gather container] 2026-03-20 16:26:48.376051 | controller | skipping: Conditional result was False 2026-03-20 16:26:48.402751 | 2026-03-20 16:26:48.402855 | TASK [os_must_gather : Set cifmw_os_must_gather_image to new image tag] 2026-03-20 16:26:48.438626 | controller | skipping: Conditional result was False 2026-03-20 16:26:48.465495 | 2026-03-20 16:26:48.465585 | TASK [os_must_gather : Check for oc command] 2026-03-20 16:26:48.867695 | controller | changed 2026-03-20 16:26:48.892293 | 2026-03-20 16:26:48.892380 | TASK [os_must_gather : Check if kubeconfig exists] 2026-03-20 16:26:49.459213 | controller | ok 2026-03-20 16:26:49.482938 | 2026-03-20 16:26:49.483027 | TASK [os_must_gather : Run openstack-must-gather command] 2026-03-20 16:32:01.589730 | controller | changed 2026-03-20 16:32:01.626011 | 2026-03-20 16:32:01.626096 | TASK [os_must_gather : Find existing os-must-gather directories] 2026-03-20 16:32:01.951146 | controller | ok: All paths examined 2026-03-20 16:32:01.978379 | 2026-03-20 16:32:01.978573 | TASK [os_must_gather : Create a symlink to newest os-must-gather directory] 2026-03-20 16:32:02.295050 | controller | changed 2026-03-20 16:32:02.312356 | 2026-03-20 16:32:02.312433 | TASK [os_must_gather : Collect pod usage] 2026-03-20 16:32:02.396153 | controller | ok 2026-03-20 16:32:02.427028 | controller | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/untrusted/project_0/github.com/openstack-k8s-operators/ci-framework/roles/os_must_gather/tasks/get_top.yml 2026-03-20 16:32:02.458040 | 2026-03-20 16:32:02.458136 | TASK [os_must_gather : Get all containers usage - sort by cpu] 2026-03-20 16:32:02.987164 | controller | error: Metrics API not available 2026-03-20 16:32:03.082856 | controller | ERROR 2026-03-20 16:32:03.083209 | controller | { 2026-03-20 16:32:03.083275 | controller | "delta": "0:00:00.171387", 2026-03-20 16:32:03.083319 | controller | "end": "2026-03-20 16:32:02.991800", 2026-03-20 16:32:03.083360 | controller | "msg": "non-zero return code", 2026-03-20 16:32:03.083398 | controller | "rc": 1, 2026-03-20 16:32:03.083444 | controller | "start": "2026-03-20 16:32:02.820413" 2026-03-20 16:32:03.083484 | controller | } 2026-03-20 16:32:03.083550 | controller | ERROR: Ignoring Errors 2026-03-20 16:32:03.127825 | 2026-03-20 16:32:03.128054 | TASK [os_must_gather : Print message when metrics are not available] 2026-03-20 16:32:03.175375 | controller | skipping: Conditional result was False 2026-03-20 16:32:03.205881 | 2026-03-20 16:32:03.206060 | TASK [os_must_gather : Get all containers usage - sort by memory] 2026-03-20 16:32:03.883475 | controller | skipping: Conditional result was False 2026-03-20 16:32:03.932035 | 2026-03-20 16:32:03.932245 | TASK [os_must_gather : Find all namespaces directories] 2026-03-20 16:32:03.972192 | controller | skipping: Conditional result was False 2026-03-20 16:32:04.017883 | 2026-03-20 16:32:04.018079 | LOOP [os_must_gather : Get resource usage by pods per namespace] 2026-03-20 16:32:04.099146 | 2026-03-20 16:32:04.099329 | TASK [os_must_gather : Get node resource usage] 2026-03-20 16:32:04.769090 | controller | skipping: Conditional result was False 2026-03-20 16:32:04.837844 | 2026-03-20 16:32:04.837985 | TASK [artifacts : Create crc logs directory] 2026-03-20 16:32:05.119053 | controller | changed 2026-03-20 16:32:05.145124 | 2026-03-20 16:32:05.145213 | TASK [artifacts : Ensure controller knows CRC ssh keys] 2026-03-20 16:32:06.271964 | controller | changed 2026-03-20 16:32:06.298517 | 2026-03-20 16:32:06.298644 | TASK [artifacts : Recognize new keypair] 2026-03-20 16:32:06.555361 | controller | ok 2026-03-20 16:32:06.572486 | 2026-03-20 16:32:06.572552 | TASK [artifacts : Set fact if new keypair exists] 2026-03-20 16:32:06.618158 | controller | skipping: Conditional result was False 2026-03-20 16:32:06.638718 | 2026-03-20 16:32:06.638810 | TASK [artifacts : Prepare root ssh accesses] 2026-03-20 16:32:10.797759 | controller | changed 2026-03-20 16:32:10.835032 | 2026-03-20 16:32:10.835143 | TASK [artifacts : Copy logs from CRC VM] 2026-03-20 16:32:13.074343 | controller | changed 2026-03-20 16:32:13.106513 | 2026-03-20 16:32:13.106605 | TASK [artifacts : List all of the existing virtual machines] 2026-03-20 16:32:13.142646 | controller | skipping: Conditional result was False 2026-03-20 16:32:13.182521 | 2026-03-20 16:32:13.182616 | TASK [artifacts : Filter out edpm vm] 2026-03-20 16:32:13.218273 | controller | skipping: Conditional result was False 2026-03-20 16:32:13.256261 | 2026-03-20 16:32:13.256365 | TASK [artifacts : Slurp zuul inventory file] 2026-03-20 16:32:13.528463 | controller | ok 2026-03-20 16:32:13.566966 | 2026-03-20 16:32:13.567111 | LOOP [artifacts : Extract Compute and Networker from zuul mapping if any] 2026-03-20 16:32:13.784470 | 2026-03-20 16:32:13.784624 | LOOP [artifacts : Generate logs on edpm vm {{ item.split('/')[0] }}] 2026-03-20 16:32:13.859389 | 2026-03-20 16:32:13.859561 | LOOP [artifacts : Copy logs to host machine from {{ item.split('/')[0] }}] 2026-03-20 16:32:13.936929 | 2026-03-20 16:32:13.937170 | TASK [artifacts : Generate list of ansible logs to collect in home directory] 2026-03-20 16:32:14.267419 | controller | ok: All paths examined 2026-03-20 16:32:14.295185 | 2026-03-20 16:32:14.295278 | LOOP [artifacts : Copy ansible logs to logs directory] 2026-03-20 16:32:14.748272 | controller | changed: 2026-03-20 16:32:14.748572 | controller | { 2026-03-20 16:32:14.748627 | controller | "atime": 1774021735.767966, 2026-03-20 16:32:14.748697 | controller | "ctime": 1774023985.9707217, 2026-03-20 16:32:14.748743 | controller | "dev": 64513, 2026-03-20 16:32:14.748796 | controller | "gid": 1000, 2026-03-20 16:32:14.748831 | controller | "gr_name": "zuul", 2026-03-20 16:32:14.748864 | controller | "inode": 1249684, 2026-03-20 16:32:14.748897 | controller | "isblk": false, 2026-03-20 16:32:14.748930 | controller | "ischr": false, 2026-03-20 16:32:14.748962 | controller | "isdir": false, 2026-03-20 16:32:14.748994 | controller | "isfifo": false, 2026-03-20 16:32:14.749026 | controller | "isgid": false, 2026-03-20 16:32:14.749059 | controller | "islnk": false, 2026-03-20 16:32:14.749101 | controller | "isreg": true, 2026-03-20 16:32:14.749135 | controller | "issock": false, 2026-03-20 16:32:14.749169 | controller | "isuid": false, 2026-03-20 16:32:14.749202 | controller | "mode": "0644", 2026-03-20 16:32:14.749235 | controller | "mtime": 1774023985.9707217, 2026-03-20 16:32:14.749268 | controller | "nlink": 1, 2026-03-20 16:32:14.749300 | controller | "path": "/home/zuul/ansible.log", 2026-03-20 16:32:14.749333 | controller | "pw_name": "zuul", 2026-03-20 16:32:14.749366 | controller | "rgrp": true, 2026-03-20 16:32:14.749398 | controller | "roth": true, 2026-03-20 16:32:14.749432 | controller | "rusr": true, 2026-03-20 16:32:14.749465 | controller | "size": 5908052, 2026-03-20 16:32:14.749498 | controller | "uid": 1000, 2026-03-20 16:32:14.749533 | controller | "wgrp": false, 2026-03-20 16:32:14.749568 | controller | "woth": false, 2026-03-20 16:32:14.749602 | controller | "wusr": true, 2026-03-20 16:32:14.749636 | controller | "xgrp": false, 2026-03-20 16:32:14.749706 | controller | "xoth": false, 2026-03-20 16:32:14.749745 | controller | "xusr": false 2026-03-20 16:32:14.749779 | controller | } 2026-03-20 16:32:14.749835 | 2026-03-20 16:32:14.798059 | 2026-03-20 16:32:14.798306 | TASK [artifacts : Ensure we have proper rights on the gathered content] 2026-03-20 16:32:16.939768 | controller | changed 2026-03-20 16:32:17.023839 | 2026-03-20 16:32:17.023990 | LOOP [artifacts : Mask secrets in yaml log files] 2026-03-20 16:34:55.568590 | controller | changed: "/home/zuul/ci-framework-data/logs" 2026-03-20 16:34:56.278856 | controller | changed: "/home/zuul/ci-framework-data/artifacts" 2026-03-20 16:34:56.355421 | 2026-03-20 16:34:56.355586 | LOOP [env_op_images : Ensure directory is present] 2026-03-20 16:34:56.624878 | controller | ok: "artifacts" 2026-03-20 16:34:56.625306 | controller | changed: All items complete 2026-03-20 16:34:56.625365 | 2026-03-20 16:34:56.804060 | controller | ok: "logs" 2026-03-20 16:34:56.888703 | 2026-03-20 16:34:56.888834 | TASK [env_op_images : Check if OpenStackControlPlane is setup] 2026-03-20 16:34:57.508144 | controller | changed 2026-03-20 16:34:57.546815 | 2026-03-20 16:34:57.547013 | TASK [env_op_images : Get images from the CSV] 2026-03-20 16:34:58.227310 | controller | skipping: Conditional result was False 2026-03-20 16:34:58.270157 | 2026-03-20 16:34:58.270304 | TASK [env_op_images : Get the images name] 2026-03-20 16:34:58.901897 | controller | skipping: Conditional result was False 2026-03-20 16:34:58.943367 | 2026-03-20 16:34:58.943511 | TASK [env_op_images : Extract env variable name and images] 2026-03-20 16:34:58.981006 | controller | skipping: Conditional result was False 2026-03-20 16:34:59.023795 | 2026-03-20 16:34:59.023888 | TASK [env_op_images : Get all pods from all namespaces to find openstack-operator-index] 2026-03-20 16:34:59.059381 | controller | skipping: Conditional result was False 2026-03-20 16:34:59.086608 | 2026-03-20 16:34:59.086721 | TASK [env_op_images : Retrieve openstack-operator-index pod] 2026-03-20 16:34:59.122106 | controller | skipping: Conditional result was False 2026-03-20 16:34:59.157278 | 2026-03-20 16:34:59.157370 | TASK [env_op_images : Get all the pods in openstack-operator namespace] 2026-03-20 16:34:59.192918 | controller | skipping: Conditional result was False 2026-03-20 16:34:59.227927 | 2026-03-20 16:34:59.228026 | TASK [env_op_images : Get operator images and pods] 2026-03-20 16:34:59.263938 | controller | skipping: Conditional result was False 2026-03-20 16:34:59.345839 | 2026-03-20 16:34:59.345971 | LOOP [env_op_images : Add operator images to the dictionary] 2026-03-20 16:34:59.408630 | 2026-03-20 16:34:59.408829 | TASK [env_op_images : Write images to file] 2026-03-20 16:34:59.444557 | controller | skipping: Conditional result was False 2026-03-20 16:34:59.498649 | 2026-03-20 16:34:59.498741 | TASK [cifmw_setup : Create a versioned log folder] 2026-03-20 16:34:59.757061 | controller | changed 2026-03-20 16:34:59.786764 | 2026-03-20 16:34:59.786882 | TASK [cifmw_setup : Return a list of log files in home directory] 2026-03-20 16:35:00.027066 | controller | ok: All paths examined 2026-03-20 16:35:00.054815 | 2026-03-20 16:35:00.054902 | TASK [cifmw_setup : Ensure ansible facts cache exists] 2026-03-20 16:35:00.308630 | controller | ok 2026-03-20 16:35:00.339592 | 2026-03-20 16:35:00.339766 | LOOP [cifmw_setup : Copy logs to proper location] 2026-03-20 16:35:00.771186 | controller | changed: 2026-03-20 16:35:00.771497 | controller | { 2026-03-20 16:35:00.771552 | controller | "atime": 1774024334.6210358, 2026-03-20 16:35:00.771591 | controller | "ctime": 1774023985.9707217, 2026-03-20 16:35:00.771627 | controller | "dev": 64513, 2026-03-20 16:35:00.771693 | controller | "gid": 1000, 2026-03-20 16:35:00.771741 | controller | "gr_name": "zuul", 2026-03-20 16:35:00.771778 | controller | "inode": 1249684, 2026-03-20 16:35:00.771812 | controller | "isblk": false, 2026-03-20 16:35:00.771845 | controller | "ischr": false, 2026-03-20 16:35:00.771878 | controller | "isdir": false, 2026-03-20 16:35:00.771912 | controller | "isfifo": false, 2026-03-20 16:35:00.771946 | controller | "isgid": false, 2026-03-20 16:35:00.771979 | controller | "islnk": false, 2026-03-20 16:35:00.772011 | controller | "isreg": true, 2026-03-20 16:35:00.772043 | controller | "issock": false, 2026-03-20 16:35:00.772074 | controller | "isuid": false, 2026-03-20 16:35:00.772126 | controller | "mode": "0644", 2026-03-20 16:35:00.772158 | controller | "mtime": 1774023985.9707217, 2026-03-20 16:35:00.772191 | controller | "nlink": 1, 2026-03-20 16:35:00.772223 | controller | "path": "/home/zuul/ansible.log", 2026-03-20 16:35:00.772255 | controller | "pw_name": "zuul", 2026-03-20 16:35:00.772288 | controller | "rgrp": true, 2026-03-20 16:35:00.772321 | controller | "roth": true, 2026-03-20 16:35:00.772355 | controller | "rusr": true, 2026-03-20 16:35:00.772387 | controller | "size": 5908052, 2026-03-20 16:35:00.772419 | controller | "uid": 1000, 2026-03-20 16:35:00.772451 | controller | "wgrp": false, 2026-03-20 16:35:00.772482 | controller | "woth": false, 2026-03-20 16:35:00.772516 | controller | "wusr": true, 2026-03-20 16:35:00.772547 | controller | "xgrp": false, 2026-03-20 16:35:00.772577 | controller | "xoth": false, 2026-03-20 16:35:00.772608 | controller | "xusr": false 2026-03-20 16:35:00.772639 | controller | } 2026-03-20 16:35:00.772784 | 2026-03-20 16:35:00.819640 | 2026-03-20 16:35:00.819861 | LOOP [cifmw_setup : Remove original log from home directory] 2026-03-20 16:35:01.087650 | controller | changed: 2026-03-20 16:35:01.087962 | controller | { 2026-03-20 16:35:01.088017 | controller | "atime": 1774024334.6210358, 2026-03-20 16:35:01.088056 | controller | "ctime": 1774023985.9707217, 2026-03-20 16:35:01.088091 | controller | "dev": 64513, 2026-03-20 16:35:01.088125 | controller | "gid": 1000, 2026-03-20 16:35:01.088158 | controller | "gr_name": "zuul", 2026-03-20 16:35:01.088190 | controller | "inode": 1249684, 2026-03-20 16:35:01.088223 | controller | "isblk": false, 2026-03-20 16:35:01.088279 | controller | "ischr": false, 2026-03-20 16:35:01.088314 | controller | "isdir": false, 2026-03-20 16:35:01.088346 | controller | "isfifo": false, 2026-03-20 16:35:01.088378 | controller | "isgid": false, 2026-03-20 16:35:01.088410 | controller | "islnk": false, 2026-03-20 16:35:01.088442 | controller | "isreg": true, 2026-03-20 16:35:01.088474 | controller | "issock": false, 2026-03-20 16:35:01.088519 | controller | "isuid": false, 2026-03-20 16:35:01.088553 | controller | "mode": "0644", 2026-03-20 16:35:01.088586 | controller | "mtime": 1774023985.9707217, 2026-03-20 16:35:01.088619 | controller | "nlink": 1, 2026-03-20 16:35:01.088652 | controller | "path": "/home/zuul/ansible.log", 2026-03-20 16:35:01.088733 | controller | "pw_name": "zuul", 2026-03-20 16:35:01.088775 | controller | "rgrp": true, 2026-03-20 16:35:01.088813 | controller | "roth": true, 2026-03-20 16:35:01.088852 | controller | "rusr": true, 2026-03-20 16:35:01.088889 | controller | "size": 5908052, 2026-03-20 16:35:01.088924 | controller | "uid": 1000, 2026-03-20 16:35:01.088958 | controller | "wgrp": false, 2026-03-20 16:35:01.088991 | controller | "woth": false, 2026-03-20 16:35:01.089024 | controller | "wusr": true, 2026-03-20 16:35:01.089057 | controller | "xgrp": false, 2026-03-20 16:35:01.089090 | controller | "xoth": false, 2026-03-20 16:35:01.089123 | controller | "xusr": false 2026-03-20 16:35:01.089155 | controller | } 2026-03-20 16:35:01.136973 | 2026-03-20 16:35:01.137164 | TASK [cifmw_setup : Copy facts to dated directory] 2026-03-20 16:35:01.584611 | controller | changed 2026-03-20 16:35:01.601914 | 2026-03-20 16:35:01.601980 | TASK [run_hook : Assert parameters are valid] 2026-03-20 16:35:01.790869 | controller | ok: All assertions passed 2026-03-20 16:35:01.815926 | 2026-03-20 16:35:01.816051 | TASK [run_hook : Assert single hooks are all mappings] 2026-03-20 16:35:01.967336 | controller | ok: All assertions passed 2026-03-20 16:35:01.993610 | 2026-03-20 16:35:01.993723 | LOOP [run_hook : Loop on hooks for post_logs] 2026-03-20 16:35:02.248926 | 2026-03-20 16:35:02.249091 | PLAY [Run ci/playbooks/e2e-collect-logs.yml on CRC host] 2026-03-20 16:35:02.270073 | 2026-03-20 16:35:02.270144 | TASK [Get kubelet journalctl logs] 2026-03-20 16:35:03.809807 | crc | Output suppressed because no_log was given 2026-03-20 16:35:03.824375 | 2026-03-20 16:35:03.824539 | TASK [Compress logs bigger than 2MB] 2026-03-20 16:35:04.890283 | crc | changed 2026-03-20 16:35:04.905370 | 2026-03-20 16:35:04.905500 | TASK [Copy files from workspace on node] 2026-03-20 16:35:04.938987 | crc | ok 2026-03-20 16:35:04.983638 | 2026-03-20 16:35:04.983769 | TASK [fetch-output : Set log path for multiple nodes] 2026-03-20 16:35:05.036594 | crc | ok 2026-03-20 16:35:05.049595 | 2026-03-20 16:35:05.049712 | TASK [fetch-output : Set log path for single node] 2026-03-20 16:35:05.075023 | crc | skipping: Conditional result was False 2026-03-20 16:35:05.085905 | 2026-03-20 16:35:05.086001 | LOOP [fetch-output : Ensure local output dirs] 2026-03-20 16:35:05.418062 | crc -> localhost | changed: "/var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/work/logs/crc" 2026-03-20 16:35:05.418595 | 2026-03-20 16:35:05.654755 | crc -> localhost | changed: "/var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/work/artifacts" 2026-03-20 16:35:05.890569 | crc -> localhost | changed: "/var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/work/docs" 2026-03-20 16:35:05.905201 | 2026-03-20 16:35:05.905311 | LOOP [fetch-output : Collect logs, artifacts and docs] 2026-03-20 16:35:06.683031 | crc | changed: 2026-03-20 16:35:06.683258 | crc | .d..t...... ./ 2026-03-20 16:35:06.683286 | crc | >f+++++++++ kubelet.log.gz 2026-03-20 16:35:07.285246 | crc | changed: .d..t...... ./ 2026-03-20 16:35:07.787605 | crc | changed: .d..t...... ./ 2026-03-20 16:35:07.836791 | 2026-03-20 16:35:07.836883 | PLAY RECAP 2026-03-20 16:35:07.836930 | controller | ok: 75 changed: 34 unreachable: 0 failed: 0 skipped: 39 rescued: 0 ignored: 2 2026-03-20 16:35:07.836955 | crc | ok: 5 changed: 4 unreachable: 0 failed: 0 skipped: 1 rescued: 0 ignored: 0 2026-03-20 16:35:07.836972 | 2026-03-20 16:35:07.999694 | POST-RUN END RESULT_NORMAL: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/e2e-collect-logs.yml@main] 2026-03-20 16:35:08.006488 | POST-RUN START: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/collect-logs.yml@main] 2026-03-20 16:35:08.602282 | 2026-03-20 16:35:08.602386 | PLAY [Run ci/playbooks/collect-logs.yml] 2026-03-20 16:35:08.712832 | 2026-03-20 16:35:08.712983 | TASK [Filter out host if needed] 2026-03-20 16:35:08.741090 | controller | skipping: end_host conditional evaluated to False, continuing execution for controller 2026-03-20 16:35:08.747962 | 2026-03-20 16:35:08.748029 | TASK [Filter out host if needed] 2026-03-20 16:35:08.780146 | 2026-03-20 16:35:08.780229 | TASK [Ensure file is present] 2026-03-20 16:35:09.252624 | controller | ok 2026-03-20 16:35:09.281256 | 2026-03-20 16:35:09.281340 | TASK [Manage molecule report file] 2026-03-20 16:35:09.969807 | controller | skipping: Conditional result was False 2026-03-20 16:35:10.014957 | 2026-03-20 16:35:10.015054 | TASK [Check if we get ci-framework-data basedir] 2026-03-20 16:35:10.306061 | controller | ok 2026-03-20 16:35:10.337376 | 2026-03-20 16:35:10.337439 | TASK [Create ci-framework-data log directory for zuul] 2026-03-20 16:35:10.790348 | controller | changed 2026-03-20 16:35:10.821008 | 2026-03-20 16:35:10.821071 | TASK [Copy ci-framework interesting files] 2026-03-20 16:35:11.445207 | controller | cp: cannot stat '/home/zuul/ci-framework-data/tests': No such file or directory 2026-03-20 16:35:22.022324 | controller | changed 2026-03-20 16:35:22.075254 | 2026-03-20 16:35:22.075381 | TASK [Get SELinux listing] 2026-03-20 16:35:32.786302 | controller | changed 2026-03-20 16:35:32.841201 | 2026-03-20 16:35:32.841437 | TASK [Generate log index] 2026-03-20 16:35:33.654614 | controller | changed 2026-03-20 16:35:33.709037 | 2026-03-20 16:35:33.709174 | TASK [Get some env related data] 2026-03-20 16:35:34.565215 | controller | /home/zuul/.local/bin/ansible 2026-03-20 16:35:45.456105 | controller | changed 2026-03-20 16:35:45.507401 | 2026-03-20 16:35:45.507496 | TASK [Generate list of logs to collect in home directory] 2026-03-20 16:35:45.905069 | controller | ok: All paths examined 2026-03-20 16:35:45.948067 | 2026-03-20 16:35:45.948284 | LOOP [Copy logs from home directory] 2026-03-20 16:35:46.011167 | 2026-03-20 16:35:46.011338 | TASK [Copy crio stats log file] 2026-03-20 16:35:46.037233 | controller | skipping: Conditional result was False 2026-03-20 16:35:46.073709 | 2026-03-20 16:35:46.073833 | TASK [Get SELinux related data] 2026-03-20 16:35:46.573573 | controller | 2026-03-20 16:35:56.824387 | controller | ERROR 2026-03-20 16:35:56.824588 | controller | { 2026-03-20 16:35:56.824620 | controller | "delta": "0:00:00.009378", 2026-03-20 16:35:56.824639 | controller | "end": "2026-03-20 16:35:46.574455", 2026-03-20 16:35:56.824658 | controller | "msg": "non-zero return code", 2026-03-20 16:35:56.824697 | controller | "rc": 1, 2026-03-20 16:35:56.824715 | controller | "start": "2026-03-20 16:35:46.565077" 2026-03-20 16:35:56.824732 | controller | } 2026-03-20 16:35:56.824754 | controller | ERROR: Ignoring Errors 2026-03-20 16:35:56.847927 | 2026-03-20 16:35:56.847992 | TASK [Create system configuration directory] 2026-03-20 16:35:57.205539 | controller | changed 2026-03-20 16:35:57.230098 | 2026-03-20 16:35:57.230162 | TASK [Get some of the system configurations] 2026-03-20 16:36:07.924134 | controller | changed 2026-03-20 16:36:07.975798 | 2026-03-20 16:36:07.975946 | TASK [Copy generated documentation if available] 2026-03-20 16:36:08.002434 | controller | skipping: Conditional result was False 2026-03-20 16:36:08.037740 | 2026-03-20 16:36:08.037868 | TASK [Copy generated AsciiDoc documentation if available] 2026-03-20 16:36:08.062862 | controller | skipping: Conditional result was False 2026-03-20 16:36:08.097071 | 2026-03-20 16:36:08.097186 | TASK [Compress logs bigger than 2MB] 2026-03-20 16:36:36.328058 | controller | changed 2026-03-20 16:36:36.364993 | 2026-03-20 16:36:36.365143 | TASK [Copy files from workspace on node] 2026-03-20 16:36:36.406026 | controller | ok 2026-03-20 16:36:36.465552 | 2026-03-20 16:36:36.465696 | TASK [fetch-output : Set log path for multiple nodes] 2026-03-20 16:36:36.525195 | controller | ok 2026-03-20 16:36:36.549374 | 2026-03-20 16:36:36.549513 | TASK [fetch-output : Set log path for single node] 2026-03-20 16:36:36.574565 | controller | skipping: Conditional result was False 2026-03-20 16:36:36.638479 | 2026-03-20 16:36:36.638618 | LOOP [fetch-output : Ensure local output dirs] 2026-03-20 16:36:36.894525 | controller -> localhost | changed: "/var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/work/logs/controller" 2026-03-20 16:36:36.894772 | 2026-03-20 16:36:37.110731 | controller -> localhost | ok: "/var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/work/artifacts" 2026-03-20 16:36:37.321881 | controller -> localhost | ok: "/var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/work/docs" 2026-03-20 16:36:37.375950 | 2026-03-20 16:36:37.376089 | LOOP [fetch-output : Collect logs, artifacts and docs] 2026-03-20 16:36:41.669945 | controller | changed: 2026-03-20 16:36:41.670186 | controller | .d..t...... ./ 2026-03-20 16:36:41.670225 | controller | >f+++++++++ README.html 2026-03-20 16:36:41.670253 | controller | >f+++++++++ dmesg.log 2026-03-20 16:36:41.670282 | controller | >f+++++++++ installed-pkgs.log 2026-03-20 16:36:41.670308 | controller | >f+++++++++ python.log 2026-03-20 16:36:41.670332 | controller | >f+++++++++ registries.conf 2026-03-20 16:36:41.670356 | controller | >f+++++++++ selinux-denials.log 2026-03-20 16:36:41.670379 | controller | >f+++++++++ selinux-listing.log 2026-03-20 16:36:41.670404 | controller | cd+++++++++ ci-framework-data/ 2026-03-20 16:36:41.670428 | controller | cd+++++++++ ci-framework-data/artifacts/ 2026-03-20 16:36:41.670452 | controller | >f+++++++++ ci-framework-data/artifacts/ansible-facts.yml 2026-03-20 16:36:41.670476 | controller | >f+++++++++ ci-framework-data/artifacts/ansible-vars.yml 2026-03-20 16:36:41.670500 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_000_check_for_oc.sh 2026-03-20 16:36:41.670523 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_000_copy_logs_from_crc.sh 2026-03-20 16:36:41.670546 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_000_prepare_root_ssh.sh 2026-03-20 16:36:41.670569 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_000_run_download.sh 2026-03-20 16:36:41.670592 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_000_run_openstack_must_gather.sh 2026-03-20 16:36:41.670614 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_001_run_hook_without_retry.sh 2026-03-20 16:36:41.670636 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_002_fetch_openshift.sh 2026-03-20 16:36:41.670659 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_003_run_openstack.sh 2026-03-20 16:36:41.670710 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_004_run_hook_without_retry_deploy.sh 2026-03-20 16:36:41.670735 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_005_stop_watcher.sh 2026-03-20 16:36:41.670757 | controller | >f+++++++++ ci-framework-data/artifacts/ci_script_006_install_watcher_operator_in.sh 2026-03-20 16:36:41.670780 | controller | >f+++++++++ ci-framework-data/artifacts/hosts 2026-03-20 16:36:41.670803 | controller | >f+++++++++ ci-framework-data/artifacts/install_yamls.sh 2026-03-20 16:36:41.670825 | controller | >f+++++++++ ci-framework-data/artifacts/installed-packages.yml 2026-03-20 16:36:41.670848 | controller | >f+++++++++ ci-framework-data/artifacts/ip-network.txt 2026-03-20 16:36:41.670871 | controller | >f+++++++++ ci-framework-data/artifacts/resolv.conf 2026-03-20 16:36:41.670894 | controller | >f+++++++++ ci-framework-data/artifacts/zuul_inventory.yml 2026-03-20 16:36:41.670917 | controller | cd+++++++++ ci-framework-data/artifacts/NetworkManager/ 2026-03-20 16:36:41.670939 | controller | >f+++++++++ ci-framework-data/artifacts/NetworkManager/ci-private-network.nmconnection 2026-03-20 16:36:41.670962 | controller | >f+++++++++ ci-framework-data/artifacts/NetworkManager/ens3.nmconnection 2026-03-20 16:36:41.670984 | controller | cd+++++++++ ci-framework-data/artifacts/ansible_facts.2026-03-20_16-35/ 2026-03-20 16:36:41.671013 | controller | cd+++++++++ ci-framework-data/artifacts/ansible_facts.2026-03-20_16-35/ansible_facts_cache/ 2026-03-20 16:36:41.671036 | controller | >f+++++++++ ci-framework-data/artifacts/ansible_facts.2026-03-20_16-35/ansible_facts_cache/controller 2026-03-20 16:36:41.671059 | controller | >f+++++++++ ci-framework-data/artifacts/ansible_facts.2026-03-20_16-35/ansible_facts_cache/localhost 2026-03-20 16:36:41.671081 | controller | cd+++++++++ ci-framework-data/artifacts/ci-env/ 2026-03-20 16:36:41.671104 | controller | >f+++++++++ ci-framework-data/artifacts/ci-env/networking-info.yml 2026-03-20 16:36:41.671127 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/ 2026-03-20 16:36:41.671153 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/cert-manager-operator/ 2026-03-20 16:36:41.671178 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/cert-manager-operator/namespace.yaml 2026-03-20 16:36:41.671202 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/cert-manager-operator/cert-manager/ 2026-03-20 16:36:41.671228 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/cert-manager-operator/cert-manager/op/ 2026-03-20 16:36:41.671252 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/cert-manager-operator/cert-manager/op/operatorgroup.yaml 2026-03-20 16:36:41.671276 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/cert-manager-operator/cert-manager/op/subscription.yaml 2026-03-20 16:36:41.671299 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/metallb-system/ 2026-03-20 16:36:41.671322 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/metallb-system/namespace.yaml 2026-03-20 16:36:41.671345 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/metallb-system/metallb/ 2026-03-20 16:36:41.671368 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/metallb-system/metallb/cr/ 2026-03-20 16:36:41.671390 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/metallb-system/metallb/cr/bgpadvertisement.yaml 2026-03-20 16:36:41.671413 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/metallb-system/metallb/cr/bgpextras.yaml 2026-03-20 16:36:41.671435 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/metallb-system/metallb/cr/bgppeers.yaml 2026-03-20 16:36:41.671457 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/metallb-system/metallb/cr/deploy_operator.yaml 2026-03-20 16:36:41.671479 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/metallb-system/metallb/cr/ipaddresspools.yaml 2026-03-20 16:36:41.671501 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/metallb-system/metallb/cr/l2advertisement.yaml 2026-03-20 16:36:41.671524 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openshift-nmstate/ 2026-03-20 16:36:41.671547 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openshift-nmstate/namespace.yaml 2026-03-20 16:36:41.671570 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openshift-nmstate/nmstate/ 2026-03-20 16:36:41.671593 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openshift-nmstate/nmstate/cr/ 2026-03-20 16:36:41.671616 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openshift-nmstate/nmstate/cr/deploy_operator.yaml 2026-03-20 16:36:41.671639 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/ 2026-03-20 16:36:41.671665 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/namespace.yaml 2026-03-20 16:36:41.671711 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/metallb/ 2026-03-20 16:36:41.671736 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/metallb/op/ 2026-03-20 16:36:41.671759 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/metallb/op/operatorgroup.yaml 2026-03-20 16:36:41.671786 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/metallb/op/subscription.yaml 2026-03-20 16:36:41.671808 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/nmstate/ 2026-03-20 16:36:41.671830 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/nmstate/op/ 2026-03-20 16:36:41.671852 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/nmstate/op/operatorgroup.yaml 2026-03-20 16:36:41.671874 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/nmstate/op/subscription.yaml 2026-03-20 16:36:41.671897 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/openstack/ 2026-03-20 16:36:41.671919 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/openstack/op/ 2026-03-20 16:36:41.671941 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/openstack/op/catalogsource.yaml 2026-03-20 16:36:41.671962 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/openstack/op/operatorgroup.yaml 2026-03-20 16:36:41.671984 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack-operators/openstack/op/subscription.yaml 2026-03-20 16:36:41.672006 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack/ 2026-03-20 16:36:41.672028 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/namespace.yaml 2026-03-20 16:36:41.672051 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack/cr/ 2026-03-20 16:36:41.672075 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/ 2026-03-20 16:36:41.672098 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/ 2026-03-20 16:36:41.672120 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/ctlplane.yaml 2026-03-20 16:36:41.672141 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/datacentre.yaml 2026-03-20 16:36:41.672163 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/designate.yaml 2026-03-20 16:36:41.672184 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/designateext.yaml 2026-03-20 16:36:41.672207 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/internalapi.yaml 2026-03-20 16:36:41.672246 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/octavia.yaml 2026-03-20 16:36:41.672270 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/storage.yaml 2026-03-20 16:36:41.672293 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/storagemgmt.yaml 2026-03-20 16:36:41.672314 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/netattach/cr/tenant.yaml 2026-03-20 16:36:41.672337 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack/nncp/ 2026-03-20 16:36:41.672359 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack/nncp/cr/ 2026-03-20 16:36:41.672381 | controller | >f+++++++++ ci-framework-data/artifacts/manifests/openstack/nncp/cr/crc_nncp.yaml 2026-03-20 16:36:41.672403 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack/openstack/ 2026-03-20 16:36:41.672426 | controller | cd+++++++++ ci-framework-data/artifacts/manifests/openstack/openstack/cr/ 2026-03-20 16:36:41.672448 | controller | cd+++++++++ ci-framework-data/artifacts/parameters/ 2026-03-20 16:36:41.672471 | controller | >f+++++++++ ci-framework-data/artifacts/parameters/custom-params.yml 2026-03-20 16:36:41.672493 | controller | >f+++++++++ ci-framework-data/artifacts/parameters/install-yamls-params.yml 2026-03-20 16:36:41.672515 | controller | >f+++++++++ ci-framework-data/artifacts/parameters/openshift-login-params.yml 2026-03-20 16:36:41.672542 | controller | >f+++++++++ ci-framework-data/artifacts/parameters/zuul-params.yml 2026-03-20 16:36:41.672565 | controller | cd+++++++++ ci-framework-data/artifacts/repositories/ 2026-03-20 16:36:41.672587 | controller | >f+++++++++ ci-framework-data/artifacts/repositories/delorean-antelope-testing.repo 2026-03-20 16:36:41.672609 | controller | >f+++++++++ ci-framework-data/artifacts/repositories/delorean.repo 2026-03-20 16:36:41.672631 | controller | >f+++++++++ ci-framework-data/artifacts/repositories/delorean.repo.md5 2026-03-20 16:36:41.672654 | controller | >f+++++++++ ci-framework-data/artifacts/repositories/repo-setup-centos-appstream.repo 2026-03-20 16:36:41.672699 | controller | >f+++++++++ ci-framework-data/artifacts/repositories/repo-setup-centos-baseos.repo 2026-03-20 16:36:41.672723 | controller | >f+++++++++ ci-framework-data/artifacts/repositories/repo-setup-centos-highavailability.repo 2026-03-20 16:36:41.672746 | controller | >f+++++++++ ci-framework-data/artifacts/repositories/repo-setup-centos-powertools.repo 2026-03-20 16:36:41.672767 | controller | cd+++++++++ ci-framework-data/artifacts/roles/ 2026-03-20 16:36:41.672790 | controller | cd+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/ 2026-03-20 16:36:41.672812 | controller | cd+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/ 2026-03-20 16:36:41.672835 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_all.yml 2026-03-20 16:36:41.672858 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_attach_default_interface.yml 2026-03-20 16:36:41.672880 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_attach_default_interface_cleanup.yml 2026-03-20 16:36:41.672903 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_barbican.yml 2026-03-20 16:36:41.672925 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_barbican_cleanup.yml 2026-03-20 16:36:41.672947 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_barbican_deploy.yml 2026-03-20 16:36:41.672968 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_barbican_deploy_cleanup.yml 2026-03-20 16:36:41.672991 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_barbican_deploy_prep.yml 2026-03-20 16:36:41.673012 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_barbican_deploy_validate.yml 2026-03-20 16:36:41.673034 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_barbican_kuttl.yml 2026-03-20 16:36:41.673055 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_barbican_kuttl_run.yml 2026-03-20 16:36:41.673077 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_barbican_prep.yml 2026-03-20 16:36:41.673099 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_baremetal.yml 2026-03-20 16:36:41.673122 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_baremetal_cleanup.yml 2026-03-20 16:36:41.673144 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_baremetal_prep.yml 2026-03-20 16:36:41.673166 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas.yml 2026-03-20 16:36:41.673188 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_baremetal_net_nad.yml 2026-03-20 16:36:41.673211 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_baremetal_net_nad_cleanup.yml 2026-03-20 16:36:41.673233 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_cleanup.yml 2026-03-20 16:36:41.673260 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_crc_attach_network.yml 2026-03-20 16:36:41.673284 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_crc_attach_network_cleanup.yml 2026-03-20 16:36:41.673306 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_crc_baremetal_bridge.yml 2026-03-20 16:36:41.673328 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_crc_baremetal_bridge_cleanup.yml 2026-03-20 16:36:41.673350 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_generate_nodes_yaml.yml 2026-03-20 16:36:41.673371 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_metallb.yml 2026-03-20 16:36:41.673394 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_metallb_cleanup.yml 2026-03-20 16:36:41.673415 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_network.yml 2026-03-20 16:36:41.673437 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_network_cleanup.yml 2026-03-20 16:36:41.673459 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_route_crc_and_crc_bmaas_networks.yml 2026-03-20 16:36:41.673481 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_route_crc_and_crc_bmaas_networks_cleanup.yml 2026-03-20 16:36:41.673503 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_sushy_emulator.yml 2026-03-20 16:36:41.673525 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_sushy_emulator_cleanup.yml 2026-03-20 16:36:41.673546 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_sushy_emulator_wait.yml 2026-03-20 16:36:41.673568 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_virtual_bms.yml 2026-03-20 16:36:41.673590 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_bmaas_virtual_bms_cleanup.yml 2026-03-20 16:36:41.673611 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ceph.yml 2026-03-20 16:36:41.673633 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ceph_cleanup.yml 2026-03-20 16:36:41.673655 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ceph_help.yml 2026-03-20 16:36:41.673705 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_certmanager.yml 2026-03-20 16:36:41.673730 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_certmanager_cleanup.yml 2026-03-20 16:36:41.673752 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cifmw_cleanup.yml 2026-03-20 16:36:41.673774 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cifmw_prepare.yml 2026-03-20 16:36:41.673894 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cinder.yml 2026-03-20 16:36:41.673929 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cinder_cleanup.yml 2026-03-20 16:36:41.673954 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cinder_deploy.yml 2026-03-20 16:36:41.673978 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cinder_deploy_cleanup.yml 2026-03-20 16:36:41.674000 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cinder_deploy_prep.yml 2026-03-20 16:36:41.674023 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cinder_kuttl.yml 2026-03-20 16:36:41.674050 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cinder_kuttl_run.yml 2026-03-20 16:36:41.674074 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cinder_prep.yml 2026-03-20 16:36:41.674096 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_cleanup.yml 2026-03-20 16:36:41.674118 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc.yml 2026-03-20 16:36:41.674140 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_attach_default_interface.yml 2026-03-20 16:36:41.674162 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_attach_default_interface_cleanup.yml 2026-03-20 16:36:41.674184 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_bmo_cleanup.yml 2026-03-20 16:36:41.674207 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_bmo_setup.yml 2026-03-20 16:36:41.674229 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_cleanup.yml 2026-03-20 16:36:41.674250 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_scrub.yml 2026-03-20 16:36:41.674272 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_storage.yml 2026-03-20 16:36:41.674297 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_storage_cleanup.yml 2026-03-20 16:36:41.674321 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_storage_cleanup_with_retries.yml 2026-03-20 16:36:41.674344 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_storage_release.yml 2026-03-20 16:36:41.674366 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_crc_storage_with_retries.yml 2026-03-20 16:36:41.674389 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_deploy_cleanup.yml 2026-03-20 16:36:41.674411 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_designate.yml 2026-03-20 16:36:41.674433 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_designate_cleanup.yml 2026-03-20 16:36:41.674456 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_designate_deploy.yml 2026-03-20 16:36:41.674479 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_designate_deploy_cleanup.yml 2026-03-20 16:36:41.674502 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_designate_deploy_prep.yml 2026-03-20 16:36:41.674525 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_designate_kuttl.yml 2026-03-20 16:36:41.674548 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_designate_kuttl_run.yml 2026-03-20 16:36:41.674570 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_designate_prep.yml 2026-03-20 16:36:41.674593 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_dns_deploy.yml 2026-03-20 16:36:41.674614 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_dns_deploy_cleanup.yml 2026-03-20 16:36:41.674636 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_dns_deploy_prep.yml 2026-03-20 16:36:41.674658 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_download_tools.yml 2026-03-20 16:36:41.674708 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_ansible_runner.yml 2026-03-20 16:36:41.674741 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_baremetal_compute.yml 2026-03-20 16:36:41.674764 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_compute.yml 2026-03-20 16:36:41.674786 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_compute_bootc.yml 2026-03-20 16:36:41.674808 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_compute_cleanup.yml 2026-03-20 16:36:41.674830 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_compute_repos.yml 2026-03-20 16:36:41.674852 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_computes_bgp.yml 2026-03-20 16:36:41.674874 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy.yml 2026-03-20 16:36:41.674897 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy_baremetal.yml 2026-03-20 16:36:41.674920 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy_baremetal_prep.yml 2026-03-20 16:36:41.674942 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy_cleanup.yml 2026-03-20 16:36:41.674964 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy_generate_keys.yml 2026-03-20 16:36:41.674987 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy_instance.yml 2026-03-20 16:36:41.675008 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy_networker.yml 2026-03-20 16:36:41.675030 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy_networker_cleanup.yml 2026-03-20 16:36:41.675052 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy_networker_prep.yml 2026-03-20 16:36:41.675074 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_deploy_prep.yml 2026-03-20 16:36:41.675096 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_networker.yml 2026-03-20 16:36:41.675118 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_networker_cleanup.yml 2026-03-20 16:36:41.675140 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_nova_discover_hosts.yml 2026-03-20 16:36:41.675162 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_patch_ansible_runner_image.yml 2026-03-20 16:36:41.675184 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_register_dns.yml 2026-03-20 16:36:41.675205 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_wait_deploy.yml 2026-03-20 16:36:41.675227 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_edpm_wait_deploy_baremetal.yml 2026-03-20 16:36:41.675250 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_glance.yml 2026-03-20 16:36:41.675272 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_glance_cleanup.yml 2026-03-20 16:36:41.675294 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_glance_deploy.yml 2026-03-20 16:36:41.675323 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_glance_deploy_cleanup.yml 2026-03-20 16:36:41.675350 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_glance_deploy_prep.yml 2026-03-20 16:36:41.675377 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_glance_kuttl.yml 2026-03-20 16:36:41.675408 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_glance_kuttl_run.yml 2026-03-20 16:36:41.675436 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_glance_prep.yml 2026-03-20 16:36:41.675458 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_heat.yml 2026-03-20 16:36:41.675480 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_heat_cleanup.yml 2026-03-20 16:36:41.675506 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_heat_deploy.yml 2026-03-20 16:36:41.675532 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_heat_deploy_cleanup.yml 2026-03-20 16:36:41.675559 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_heat_deploy_prep.yml 2026-03-20 16:36:41.675586 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_heat_kuttl.yml 2026-03-20 16:36:41.675612 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_heat_kuttl_crc.yml 2026-03-20 16:36:41.675637 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_heat_kuttl_run.yml 2026-03-20 16:36:41.675680 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_heat_prep.yml 2026-03-20 16:36:41.675782 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_help.yml 2026-03-20 16:36:41.675812 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_horizon.yml 2026-03-20 16:36:41.675836 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_horizon_cleanup.yml 2026-03-20 16:36:41.675859 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_horizon_deploy.yml 2026-03-20 16:36:41.675882 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_horizon_deploy_cleanup.yml 2026-03-20 16:36:41.675904 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_horizon_deploy_prep.yml 2026-03-20 16:36:41.675926 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_horizon_kuttl.yml 2026-03-20 16:36:41.675947 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_horizon_kuttl_run.yml 2026-03-20 16:36:41.675968 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_horizon_prep.yml 2026-03-20 16:36:41.675990 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_infra.yml 2026-03-20 16:36:41.676011 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_infra_cleanup.yml 2026-03-20 16:36:41.676033 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_infra_kuttl.yml 2026-03-20 16:36:41.676055 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_infra_kuttl_run.yml 2026-03-20 16:36:41.676076 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_infra_prep.yml 2026-03-20 16:36:41.676096 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_input.yml 2026-03-20 16:36:41.676117 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_input_cleanup.yml 2026-03-20 16:36:41.676138 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ipv6_lab.yml 2026-03-20 16:36:41.676160 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ipv6_lab_cleanup.yml 2026-03-20 16:36:41.676182 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ipv6_lab_nat64_router.yml 2026-03-20 16:36:41.676209 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ipv6_lab_nat64_router_cleanup.yml 2026-03-20 16:36:41.676232 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ipv6_lab_network.yml 2026-03-20 16:36:41.676275 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ipv6_lab_network_cleanup.yml 2026-03-20 16:36:41.676299 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ipv6_lab_sno.yml 2026-03-20 16:36:41.676322 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ipv6_lab_sno_cleanup.yml 2026-03-20 16:36:41.676344 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ironic.yml 2026-03-20 16:36:41.676366 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ironic_cleanup.yml 2026-03-20 16:36:41.676388 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ironic_deploy.yml 2026-03-20 16:36:41.676410 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ironic_deploy_cleanup.yml 2026-03-20 16:36:41.676432 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ironic_deploy_prep.yml 2026-03-20 16:36:41.676455 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ironic_kuttl.yml 2026-03-20 16:36:41.676477 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ironic_kuttl_crc.yml 2026-03-20 16:36:41.676499 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ironic_kuttl_run.yml 2026-03-20 16:36:41.676521 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ironic_prep.yml 2026-03-20 16:36:41.676543 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_keystone.yml 2026-03-20 16:36:41.676565 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_keystone_cleanup.yml 2026-03-20 16:36:41.676588 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_keystone_deploy.yml 2026-03-20 16:36:41.676611 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_keystone_deploy_cleanup.yml 2026-03-20 16:36:41.676634 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_keystone_deploy_prep.yml 2026-03-20 16:36:41.676656 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_keystone_kuttl.yml 2026-03-20 16:36:41.676710 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_keystone_kuttl_run.yml 2026-03-20 16:36:41.676736 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_keystone_prep.yml 2026-03-20 16:36:41.676758 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_kuttl_common_cleanup.yml 2026-03-20 16:36:41.676781 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_kuttl_common_prep.yml 2026-03-20 16:36:41.676803 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_kuttl_db_cleanup.yml 2026-03-20 16:36:41.676825 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_kuttl_db_prep.yml 2026-03-20 16:36:41.676850 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_loki.yml 2026-03-20 16:36:41.676877 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_loki_cleanup.yml 2026-03-20 16:36:41.676901 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_loki_deploy.yml 2026-03-20 16:36:41.676930 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_loki_deploy_cleanup.yml 2026-03-20 16:36:41.676954 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_lvms.yml 2026-03-20 16:36:41.676977 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_manila.yml 2026-03-20 16:36:41.677000 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_manila_cleanup.yml 2026-03-20 16:36:41.677023 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_manila_deploy.yml 2026-03-20 16:36:41.677046 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_manila_deploy_cleanup.yml 2026-03-20 16:36:41.677069 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_manila_deploy_prep.yml 2026-03-20 16:36:41.677092 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_manila_kuttl.yml 2026-03-20 16:36:41.677115 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_manila_kuttl_run.yml 2026-03-20 16:36:41.677138 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_manila_prep.yml 2026-03-20 16:36:41.677162 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mariadb.yml 2026-03-20 16:36:41.677185 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mariadb_chainsaw.yml 2026-03-20 16:36:41.677208 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mariadb_chainsaw_run.yml 2026-03-20 16:36:41.677231 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mariadb_cleanup.yml 2026-03-20 16:36:41.677254 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mariadb_deploy.yml 2026-03-20 16:36:41.677277 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mariadb_deploy_cleanup.yml 2026-03-20 16:36:41.677300 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mariadb_deploy_prep.yml 2026-03-20 16:36:41.677323 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mariadb_kuttl.yml 2026-03-20 16:36:41.677346 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mariadb_kuttl_run.yml 2026-03-20 16:36:41.677369 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_memcached_deploy.yml 2026-03-20 16:36:41.677392 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_memcached_deploy_cleanup.yml 2026-03-20 16:36:41.677415 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_memcached_deploy_prep.yml 2026-03-20 16:36:41.677438 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_metallb.yml 2026-03-20 16:36:41.677461 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_metallb_cleanup.yml 2026-03-20 16:36:41.677485 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_metallb_config.yml 2026-03-20 16:36:41.677508 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_metallb_config_cleanup.yml 2026-03-20 16:36:41.677532 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mirror_registry.yml 2026-03-20 16:36:41.677555 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mirror_registry_ca.yml 2026-03-20 16:36:41.677578 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mirror_registry_cleanup.yml 2026-03-20 16:36:41.677605 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_mirror_registry_secure.yml 2026-03-20 16:36:41.677629 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_namespace.yml 2026-03-20 16:36:41.677652 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_namespace_cleanup.yml 2026-03-20 16:36:41.677703 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_netattach.yml 2026-03-20 16:36:41.677728 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_netattach_cleanup.yml 2026-03-20 16:36:41.677752 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_netconfig_deploy.yml 2026-03-20 16:36:41.677775 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_netconfig_deploy_cleanup.yml 2026-03-20 16:36:41.677799 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_netconfig_deploy_prep.yml 2026-03-20 16:36:41.677823 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_netobserv.yml 2026-03-20 16:36:41.677847 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_netobserv_cleanup.yml 2026-03-20 16:36:41.677871 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_netobserv_deploy.yml 2026-03-20 16:36:41.677896 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_netobserv_deploy_cleanup.yml 2026-03-20 16:36:41.677921 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_network_isolation_bridge.yml 2026-03-20 16:36:41.677946 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_network_isolation_bridge_cleanup.yml 2026-03-20 16:36:41.677970 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_neutron.yml 2026-03-20 16:36:41.677995 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_neutron_cleanup.yml 2026-03-20 16:36:41.678018 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_neutron_deploy.yml 2026-03-20 16:36:41.678042 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_neutron_deploy_cleanup.yml 2026-03-20 16:36:41.678065 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_neutron_deploy_prep.yml 2026-03-20 16:36:41.678088 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_neutron_kuttl.yml 2026-03-20 16:36:41.678110 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_neutron_kuttl_run.yml 2026-03-20 16:36:41.678133 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_neutron_prep.yml 2026-03-20 16:36:41.678155 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nfs.yml 2026-03-20 16:36:41.678178 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nfs_cleanup.yml 2026-03-20 16:36:41.678201 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nmstate.yml 2026-03-20 16:36:41.678224 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nncp.yml 2026-03-20 16:36:41.678248 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nncp_cleanup.yml 2026-03-20 16:36:41.678271 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nova.yml 2026-03-20 16:36:41.678295 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nova_cleanup.yml 2026-03-20 16:36:41.678319 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nova_deploy.yml 2026-03-20 16:36:41.678347 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nova_deploy_cleanup.yml 2026-03-20 16:36:41.678371 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nova_deploy_prep.yml 2026-03-20 16:36:41.678394 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_nova_prep.yml 2026-03-20 16:36:41.678417 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_octavia.yml 2026-03-20 16:36:41.678440 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_octavia_cleanup.yml 2026-03-20 16:36:41.678463 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_octavia_deploy.yml 2026-03-20 16:36:41.678486 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_octavia_deploy_cleanup.yml 2026-03-20 16:36:41.678509 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_octavia_deploy_prep.yml 2026-03-20 16:36:41.678532 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_octavia_kuttl.yml 2026-03-20 16:36:41.678554 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_octavia_kuttl_run.yml 2026-03-20 16:36:41.678577 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_octavia_prep.yml 2026-03-20 16:36:41.678600 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack.yml 2026-03-20 16:36:41.678624 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_cleanup.yml 2026-03-20 16:36:41.678646 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_crds.yml 2026-03-20 16:36:41.678696 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_crds_cleanup.yml 2026-03-20 16:36:41.678722 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_deploy.yml 2026-03-20 16:36:41.678746 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_deploy_cleanup.yml 2026-03-20 16:36:41.678769 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_deploy_prep.yml 2026-03-20 16:36:41.678792 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_init.yml 2026-03-20 16:36:41.678815 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_kuttl.yml 2026-03-20 16:36:41.678838 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_kuttl_cleanup.yml 2026-03-20 16:36:41.678861 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_kuttl_prep.yml 2026-03-20 16:36:41.678884 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_kuttl_run.yml 2026-03-20 16:36:41.678907 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_patch_version.yml 2026-03-20 16:36:41.678930 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_prep.yml 2026-03-20 16:36:41.678953 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_repo.yml 2026-03-20 16:36:41.678976 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_update_run.yml 2026-03-20 16:36:41.678999 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_wait.yml 2026-03-20 16:36:41.679022 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_openstack_wait_deploy.yml 2026-03-20 16:36:41.679050 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_operator_namespace.yml 2026-03-20 16:36:41.679074 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ovn.yml 2026-03-20 16:36:41.679098 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ovn_cleanup.yml 2026-03-20 16:36:41.679121 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ovn_deploy.yml 2026-03-20 16:36:41.679144 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ovn_deploy_cleanup.yml 2026-03-20 16:36:41.679168 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ovn_deploy_prep.yml 2026-03-20 16:36:41.679191 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ovn_kuttl.yml 2026-03-20 16:36:41.679214 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ovn_kuttl_run.yml 2026-03-20 16:36:41.679237 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_ovn_prep.yml 2026-03-20 16:36:41.679260 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_placement.yml 2026-03-20 16:36:41.679283 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_placement_cleanup.yml 2026-03-20 16:36:41.679307 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_placement_deploy.yml 2026-03-20 16:36:41.679330 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_placement_deploy_cleanup.yml 2026-03-20 16:36:41.679353 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_placement_deploy_prep.yml 2026-03-20 16:36:41.679375 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_placement_kuttl.yml 2026-03-20 16:36:41.679398 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_placement_kuttl_run.yml 2026-03-20 16:36:41.679421 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_placement_prep.yml 2026-03-20 16:36:41.679444 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rabbitmq.yml 2026-03-20 16:36:41.679467 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rabbitmq_cleanup.yml 2026-03-20 16:36:41.679491 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rabbitmq_deploy.yml 2026-03-20 16:36:41.679515 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rabbitmq_deploy_cleanup.yml 2026-03-20 16:36:41.679538 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rabbitmq_deploy_prep.yml 2026-03-20 16:36:41.679562 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rabbitmq_prep.yml 2026-03-20 16:36:41.679587 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_redis_deploy.yml 2026-03-20 16:36:41.679610 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_redis_deploy_cleanup.yml 2026-03-20 16:36:41.679634 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_redis_deploy_prep.yml 2026-03-20 16:36:41.679656 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rook.yml 2026-03-20 16:36:41.679708 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rook_cleanup.yml 2026-03-20 16:36:41.679734 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rook_crc_disk.yml 2026-03-20 16:36:41.679758 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rook_deploy.yml 2026-03-20 16:36:41.679786 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rook_deploy_prep.yml 2026-03-20 16:36:41.679811 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_rook_prep.yml 2026-03-20 16:36:41.679834 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_set_slower_etcd_profile.yml 2026-03-20 16:36:41.679857 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_standalone.yml 2026-03-20 16:36:41.679881 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_standalone_cleanup.yml 2026-03-20 16:36:41.679904 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_standalone_deploy.yml 2026-03-20 16:36:41.679928 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_standalone_revert.yml 2026-03-20 16:36:41.679952 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_standalone_snapshot.yml 2026-03-20 16:36:41.679976 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_standalone_sync.yml 2026-03-20 16:36:41.679999 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_swift.yml 2026-03-20 16:36:41.680023 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_swift_cleanup.yml 2026-03-20 16:36:41.680046 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_swift_deploy.yml 2026-03-20 16:36:41.680070 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_swift_deploy_cleanup.yml 2026-03-20 16:36:41.680092 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_swift_deploy_prep.yml 2026-03-20 16:36:41.680116 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_swift_kuttl.yml 2026-03-20 16:36:41.680139 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_swift_kuttl_run.yml 2026-03-20 16:36:41.680161 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_swift_prep.yml 2026-03-20 16:36:41.680184 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_telemetry.yml 2026-03-20 16:36:41.680207 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_telemetry_cleanup.yml 2026-03-20 16:36:41.680229 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_telemetry_deploy.yml 2026-03-20 16:36:41.680252 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_telemetry_deploy_cleanup.yml 2026-03-20 16:36:41.680295 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_telemetry_deploy_prep.yml 2026-03-20 16:36:41.680319 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_telemetry_kuttl.yml 2026-03-20 16:36:41.680342 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_telemetry_kuttl_run.yml 2026-03-20 16:36:41.680365 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_telemetry_prep.yml 2026-03-20 16:36:41.680388 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_tripleo_deploy.yml 2026-03-20 16:36:41.680411 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_update_services.yml 2026-03-20 16:36:41.680434 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_update_system.yml 2026-03-20 16:36:41.680456 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_validate_marketplace.yml 2026-03-20 16:36:41.680485 | controller | >f+++++++++ ci-framework-data/artifacts/roles/install_yamls_makes/tasks/make_wait.yml 2026-03-20 16:36:41.680508 | controller | cd+++++++++ ci-framework-data/artifacts/yum_repos/ 2026-03-20 16:36:41.680535 | controller | >f+++++++++ ci-framework-data/artifacts/yum_repos/delorean-antelope-testing.repo 2026-03-20 16:36:41.680561 | controller | >f+++++++++ ci-framework-data/artifacts/yum_repos/delorean.repo 2026-03-20 16:36:41.680587 | controller | >f+++++++++ ci-framework-data/artifacts/yum_repos/delorean.repo.md5 2026-03-20 16:36:41.680611 | controller | >f+++++++++ ci-framework-data/artifacts/yum_repos/repo-setup-centos-appstream.repo 2026-03-20 16:36:41.680636 | controller | >f+++++++++ ci-framework-data/artifacts/yum_repos/repo-setup-centos-baseos.repo 2026-03-20 16:36:41.680660 | controller | >f+++++++++ ci-framework-data/artifacts/yum_repos/repo-setup-centos-highavailability.repo 2026-03-20 16:36:41.680709 | controller | >f+++++++++ ci-framework-data/artifacts/yum_repos/repo-setup-centos-powertools.repo 2026-03-20 16:36:41.680735 | controller | cd+++++++++ ci-framework-data/logs/ 2026-03-20 16:36:41.680760 | controller | >f+++++++++ ci-framework-data/logs/ansible.log.gz 2026-03-20 16:36:41.680784 | controller | >f+++++++++ ci-framework-data/logs/ci_script_000_copy_logs_from_crc.log 2026-03-20 16:36:41.680808 | controller | >f+++++++++ ci-framework-data/logs/ci_script_000_prepare_root_ssh.log 2026-03-20 16:36:41.680833 | controller | >f+++++++++ ci-framework-data/logs/ci_script_000_run_download.log 2026-03-20 16:36:41.680857 | controller | >f+++++++++ ci-framework-data/logs/ci_script_000_run_openstack_must_gather.log 2026-03-20 16:36:41.680882 | controller | >f+++++++++ ci-framework-data/logs/ci_script_001_run_hook_without_retry.log 2026-03-20 16:36:41.680906 | controller | >f+++++++++ ci-framework-data/logs/ci_script_002_fetch_openshift.log 2026-03-20 16:36:41.680929 | controller | >f+++++++++ ci-framework-data/logs/ci_script_003_run_openstack.log 2026-03-20 16:36:41.680953 | controller | >f+++++++++ ci-framework-data/logs/ci_script_004_run_hook_without_retry_deploy.log 2026-03-20 16:36:41.680977 | controller | >f+++++++++ ci-framework-data/logs/ci_script_005_stop_watcher.log 2026-03-20 16:36:41.681000 | controller | >f+++++++++ ci-framework-data/logs/ci_script_006_install_watcher_operator_in.log 2026-03-20 16:36:41.681024 | controller | >f+++++++++ ci-framework-data/logs/post_install_operators_kuttl_from_operator_deploy_watcher_operator.log 2026-03-20 16:36:41.681047 | controller | >f+++++++++ ci-framework-data/logs/pre_infra_download_needed_tools.log 2026-03-20 16:36:41.681071 | controller | cd+++++++++ ci-framework-data/logs/2026-03-20_16-26/ 2026-03-20 16:36:41.681095 | controller | >f+++++++++ ci-framework-data/logs/2026-03-20_16-26/ansible.log.gz 2026-03-20 16:36:41.681118 | controller | cd+++++++++ ci-framework-data/logs/crc/ 2026-03-20 16:36:41.681142 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/ 2026-03-20 16:36:41.681165 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/ 2026-03-20 16:36:41.681190 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager-operator_cert-manager-operator-controller-manager-66c8bdd694-54nlc_0bd5f72b-6f44-4aeb-b0c7-e8bc3d2821ab/ 2026-03-20 16:36:41.681215 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager-operator_cert-manager-operator-controller-manager-66c8bdd694-54nlc_0bd5f72b-6f44-4aeb-b0c7-e8bc3d2821ab/cert-manager-operator/ 2026-03-20 16:36:41.681240 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager-operator_cert-manager-operator-controller-manager-66c8bdd694-54nlc_0bd5f72b-6f44-4aeb-b0c7-e8bc3d2821ab/cert-manager-operator/0.log 2026-03-20 16:36:41.681265 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager_cert-manager-545d4d4674-7rt5m_1de6b68c-a264-4e69-8f1d-5427473d0b31/ 2026-03-20 16:36:41.681294 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager_cert-manager-545d4d4674-7rt5m_1de6b68c-a264-4e69-8f1d-5427473d0b31/cert-manager-controller/ 2026-03-20 16:36:41.681318 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager_cert-manager-545d4d4674-7rt5m_1de6b68c-a264-4e69-8f1d-5427473d0b31/cert-manager-controller/0.log 2026-03-20 16:36:41.681342 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager_cert-manager-cainjector-5545bd876-st7jn_3abcfa4b-b69a-41ea-83e2-589697723bef/ 2026-03-20 16:36:41.681366 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager_cert-manager-cainjector-5545bd876-st7jn_3abcfa4b-b69a-41ea-83e2-589697723bef/cert-manager-cainjector/ 2026-03-20 16:36:41.681389 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager_cert-manager-cainjector-5545bd876-st7jn_3abcfa4b-b69a-41ea-83e2-589697723bef/cert-manager-cainjector/0.log 2026-03-20 16:36:41.681412 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager_cert-manager-webhook-6888856db4-n56d9_d9ff7c1b-30f8-49bf-b626-3911d0943736/ 2026-03-20 16:36:41.681435 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager_cert-manager-webhook-6888856db4-n56d9_d9ff7c1b-30f8-49bf-b626-3911d0943736/cert-manager-webhook/ 2026-03-20 16:36:41.681458 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/cert-manager_cert-manager-webhook-6888856db4-n56d9_d9ff7c1b-30f8-49bf-b626-3911d0943736/cert-manager-webhook/0.log 2026-03-20 16:36:41.681480 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/hostpath-provisioner_csi-hostpathplugin-m4mtq_e41db8d8-3508-40dc-bca8-ecd9ce41a6e3/ 2026-03-20 16:36:41.681503 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/hostpath-provisioner_csi-hostpathplugin-m4mtq_e41db8d8-3508-40dc-bca8-ecd9ce41a6e3/csi-provisioner/ 2026-03-20 16:36:41.681526 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/hostpath-provisioner_csi-hostpathplugin-m4mtq_e41db8d8-3508-40dc-bca8-ecd9ce41a6e3/csi-provisioner/0.log 2026-03-20 16:36:41.681548 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/hostpath-provisioner_csi-hostpathplugin-m4mtq_e41db8d8-3508-40dc-bca8-ecd9ce41a6e3/hostpath-provisioner/ 2026-03-20 16:36:41.681571 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/hostpath-provisioner_csi-hostpathplugin-m4mtq_e41db8d8-3508-40dc-bca8-ecd9ce41a6e3/hostpath-provisioner/0.log 2026-03-20 16:36:41.681594 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/hostpath-provisioner_csi-hostpathplugin-m4mtq_e41db8d8-3508-40dc-bca8-ecd9ce41a6e3/liveness-probe/ 2026-03-20 16:36:41.681618 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/hostpath-provisioner_csi-hostpathplugin-m4mtq_e41db8d8-3508-40dc-bca8-ecd9ce41a6e3/liveness-probe/0.log 2026-03-20 16:36:41.681642 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/hostpath-provisioner_csi-hostpathplugin-m4mtq_e41db8d8-3508-40dc-bca8-ecd9ce41a6e3/node-driver-registrar/ 2026-03-20 16:36:41.681686 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/hostpath-provisioner_csi-hostpathplugin-m4mtq_e41db8d8-3508-40dc-bca8-ecd9ce41a6e3/node-driver-registrar/0.log 2026-03-20 16:36:41.681717 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_controller-7bb4cc7c98-f45rt_c9ca7072-6004-41c6-8090-d5bd3369994b/ 2026-03-20 16:36:41.681742 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_controller-7bb4cc7c98-f45rt_c9ca7072-6004-41c6-8090-d5bd3369994b/controller/ 2026-03-20 16:36:41.681766 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_controller-7bb4cc7c98-f45rt_c9ca7072-6004-41c6-8090-d5bd3369994b/controller/0.log.gz 2026-03-20 16:36:41.681795 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_controller-7bb4cc7c98-f45rt_c9ca7072-6004-41c6-8090-d5bd3369994b/kube-rbac-proxy/ 2026-03-20 16:36:41.681820 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_controller-7bb4cc7c98-f45rt_c9ca7072-6004-41c6-8090-d5bd3369994b/kube-rbac-proxy/0.log 2026-03-20 16:36:41.681844 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-fbdp4_090f54f0-8cbc-4ecf-b792-c160a26595ff/ 2026-03-20 16:36:41.681870 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-fbdp4_090f54f0-8cbc-4ecf-b792-c160a26595ff/5f32618edfa18e2b67797ab88323bc320e6092c37a622dd1728c81bd9b7c9f3e.log 2026-03-20 16:36:41.681894 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-fbdp4_090f54f0-8cbc-4ecf-b792-c160a26595ff/controller/ 2026-03-20 16:36:41.681918 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-fbdp4_090f54f0-8cbc-4ecf-b792-c160a26595ff/controller/0.log 2026-03-20 16:36:41.681941 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-fbdp4_090f54f0-8cbc-4ecf-b792-c160a26595ff/cp-frr-files/ 2026-03-20 16:36:41.681965 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-fbdp4_090f54f0-8cbc-4ecf-b792-c160a26595ff/cp-frr-files/0.log 2026-03-20 16:36:41.681988 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-fbdp4_090f54f0-8cbc-4ecf-b792-c160a26595ff/cp-metrics/ 2026-03-20 16:36:41.682011 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-fbdp4_090f54f0-8cbc-4ecf-b792-c160a26595ff/cp-metrics/0.log 2026-03-20 16:36:41.682034 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-fbdp4_090f54f0-8cbc-4ecf-b792-c160a26595ff/cp-reloader/ 2026-03-20 16:36:41.682057 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-fbdp4_090f54f0-8cbc-4ecf-b792-c160a26595ff/cp-reloader/0.log 2026-03-20 16:36:41.682079 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-fbdp4_090f54f0-8cbc-4ecf-b792-c160a26595ff/frr-metrics/ 2026-03-20 16:36:41.682102 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-fbdp4_090f54f0-8cbc-4ecf-b792-c160a26595ff/frr-metrics/0.log 2026-03-20 16:36:41.682125 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-fbdp4_090f54f0-8cbc-4ecf-b792-c160a26595ff/frr/ 2026-03-20 16:36:41.682148 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-fbdp4_090f54f0-8cbc-4ecf-b792-c160a26595ff/frr/0.log.gz 2026-03-20 16:36:41.682171 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-fbdp4_090f54f0-8cbc-4ecf-b792-c160a26595ff/kube-rbac-proxy-frr/ 2026-03-20 16:36:41.682193 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-fbdp4_090f54f0-8cbc-4ecf-b792-c160a26595ff/kube-rbac-proxy-frr/0.log 2026-03-20 16:36:41.682216 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-fbdp4_090f54f0-8cbc-4ecf-b792-c160a26595ff/kube-rbac-proxy/ 2026-03-20 16:36:41.682239 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-fbdp4_090f54f0-8cbc-4ecf-b792-c160a26595ff/kube-rbac-proxy/0.log 2026-03-20 16:36:41.682261 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-fbdp4_090f54f0-8cbc-4ecf-b792-c160a26595ff/reloader/ 2026-03-20 16:36:41.682284 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-fbdp4_090f54f0-8cbc-4ecf-b792-c160a26595ff/reloader/0.log 2026-03-20 16:36:41.682311 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-webhook-server-bcc4b6f68-fxssk_5d96ab75-0ca7-47bc-b85d-b39649c76561/ 2026-03-20 16:36:41.682334 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-webhook-server-bcc4b6f68-fxssk_5d96ab75-0ca7-47bc-b85d-b39649c76561/frr-k8s-webhook-server/ 2026-03-20 16:36:41.682356 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_frr-k8s-webhook-server-bcc4b6f68-fxssk_5d96ab75-0ca7-47bc-b85d-b39649c76561/frr-k8s-webhook-server/0.log 2026-03-20 16:36:41.682379 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_metallb-operator-controller-manager-bfc448998-db8w6_28b6ff98-fd32-44bd-8ab0-9395fa76e6f8/ 2026-03-20 16:36:41.682402 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_metallb-operator-controller-manager-bfc448998-db8w6_28b6ff98-fd32-44bd-8ab0-9395fa76e6f8/manager/ 2026-03-20 16:36:41.682424 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_metallb-operator-controller-manager-bfc448998-db8w6_28b6ff98-fd32-44bd-8ab0-9395fa76e6f8/manager/0.log 2026-03-20 16:36:41.682447 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_metallb-operator-webhook-server-795784bfcc-vcd2s_5673dcee-fdfe-4f75-9dca-b35a64c13bea/ 2026-03-20 16:36:41.682469 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_metallb-operator-webhook-server-795784bfcc-vcd2s_5673dcee-fdfe-4f75-9dca-b35a64c13bea/webhook-server/ 2026-03-20 16:36:41.682492 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_metallb-operator-webhook-server-795784bfcc-vcd2s_5673dcee-fdfe-4f75-9dca-b35a64c13bea/webhook-server/0.log 2026-03-20 16:36:41.682515 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_speaker-skmm6_b9924c18-4463-4445-a18f-5bd9e6ec1334/ 2026-03-20 16:36:41.682539 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_speaker-skmm6_b9924c18-4463-4445-a18f-5bd9e6ec1334/kube-rbac-proxy/ 2026-03-20 16:36:41.682562 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_speaker-skmm6_b9924c18-4463-4445-a18f-5bd9e6ec1334/kube-rbac-proxy/0.log 2026-03-20 16:36:41.682586 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_speaker-skmm6_b9924c18-4463-4445-a18f-5bd9e6ec1334/speaker/ 2026-03-20 16:36:41.682609 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/metallb-system_speaker-skmm6_b9924c18-4463-4445-a18f-5bd9e6ec1334/speaker/0.log.gz 2026-03-20 16:36:41.682633 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver-operator_openshift-apiserver-operator-796bbdcf4f-x2rnw_c099999d-9e76-4625-ad9a-2a9d0d293a63/ 2026-03-20 16:36:41.682656 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver-operator_openshift-apiserver-operator-796bbdcf4f-x2rnw_c099999d-9e76-4625-ad9a-2a9d0d293a63/openshift-apiserver-operator/ 2026-03-20 16:36:41.682707 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver-operator_openshift-apiserver-operator-796bbdcf4f-x2rnw_c099999d-9e76-4625-ad9a-2a9d0d293a63/openshift-apiserver-operator/0.log 2026-03-20 16:36:41.682733 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver_apiserver-76f77b778f-h2fdd_102c2351-cd99-4355-b31f-ac1fff221c48/ 2026-03-20 16:36:41.682757 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver_apiserver-76f77b778f-h2fdd_102c2351-cd99-4355-b31f-ac1fff221c48/fix-audit-permissions/ 2026-03-20 16:36:41.682786 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver_apiserver-76f77b778f-h2fdd_102c2351-cd99-4355-b31f-ac1fff221c48/fix-audit-permissions/0.log 2026-03-20 16:36:41.682810 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver_apiserver-76f77b778f-h2fdd_102c2351-cd99-4355-b31f-ac1fff221c48/openshift-apiserver-check-endpoints/ 2026-03-20 16:36:41.682833 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver_apiserver-76f77b778f-h2fdd_102c2351-cd99-4355-b31f-ac1fff221c48/openshift-apiserver-check-endpoints/0.log 2026-03-20 16:36:41.682856 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver_apiserver-76f77b778f-h2fdd_102c2351-cd99-4355-b31f-ac1fff221c48/openshift-apiserver/ 2026-03-20 16:36:41.682879 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-apiserver_apiserver-76f77b778f-h2fdd_102c2351-cd99-4355-b31f-ac1fff221c48/openshift-apiserver/0.log 2026-03-20 16:36:41.682903 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-authentication-operator_authentication-operator-69f744f599-llrbn_982e4ba6-24e6-4165-9bae-c805735078e0/ 2026-03-20 16:36:41.682926 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-authentication-operator_authentication-operator-69f744f599-llrbn_982e4ba6-24e6-4165-9bae-c805735078e0/authentication-operator/ 2026-03-20 16:36:41.682950 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-authentication-operator_authentication-operator-69f744f599-llrbn_982e4ba6-24e6-4165-9bae-c805735078e0/authentication-operator/0.log 2026-03-20 16:36:41.682974 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-authentication_oauth-openshift-78779c56d4-vqtcw_bf5fbd2d-39bf-4642-9ada-14cf7e141542/ 2026-03-20 16:36:41.682998 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-authentication_oauth-openshift-78779c56d4-vqtcw_bf5fbd2d-39bf-4642-9ada-14cf7e141542/oauth-openshift/ 2026-03-20 16:36:41.683022 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-authentication_oauth-openshift-78779c56d4-vqtcw_bf5fbd2d-39bf-4642-9ada-14cf7e141542/oauth-openshift/0.log 2026-03-20 16:36:41.683045 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-machine-approver_machine-approver-56656f9798-5n7dz_2e4ed81d-1d4c-4bd2-8ceb-4e959b8a7fb1/ 2026-03-20 16:36:41.683068 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-machine-approver_machine-approver-56656f9798-5n7dz_2e4ed81d-1d4c-4bd2-8ceb-4e959b8a7fb1/kube-rbac-proxy/ 2026-03-20 16:36:41.683091 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-machine-approver_machine-approver-56656f9798-5n7dz_2e4ed81d-1d4c-4bd2-8ceb-4e959b8a7fb1/kube-rbac-proxy/0.log 2026-03-20 16:36:41.683114 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-machine-approver_machine-approver-56656f9798-5n7dz_2e4ed81d-1d4c-4bd2-8ceb-4e959b8a7fb1/machine-approver-controller/ 2026-03-20 16:36:41.683138 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-machine-approver_machine-approver-56656f9798-5n7dz_2e4ed81d-1d4c-4bd2-8ceb-4e959b8a7fb1/machine-approver-controller/0.log 2026-03-20 16:36:41.683161 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-samples-operator_cluster-samples-operator-665b6dd947-f528s_2c9ef6af-6946-4a87-9b7f-1ff56f5ea780/ 2026-03-20 16:36:41.683184 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-samples-operator_cluster-samples-operator-665b6dd947-f528s_2c9ef6af-6946-4a87-9b7f-1ff56f5ea780/715c74b108471b13d205c35f097cd33eeaffe7ec5ddbe847acac46652f2e6723.log 2026-03-20 16:36:41.683213 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-samples-operator_cluster-samples-operator-665b6dd947-f528s_2c9ef6af-6946-4a87-9b7f-1ff56f5ea780/cluster-samples-operator-watch/ 2026-03-20 16:36:41.683237 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-samples-operator_cluster-samples-operator-665b6dd947-f528s_2c9ef6af-6946-4a87-9b7f-1ff56f5ea780/cluster-samples-operator-watch/0.log 2026-03-20 16:36:41.683262 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-samples-operator_cluster-samples-operator-665b6dd947-f528s_2c9ef6af-6946-4a87-9b7f-1ff56f5ea780/cluster-samples-operator/ 2026-03-20 16:36:41.683287 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-samples-operator_cluster-samples-operator-665b6dd947-f528s_2c9ef6af-6946-4a87-9b7f-1ff56f5ea780/cluster-samples-operator/0.log 2026-03-20 16:36:41.683312 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-version_cluster-version-operator-5c965bbfc6-9gtdt_fde7973f-1003-4839-a94c-91f713c5719b/ 2026-03-20 16:36:41.683336 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-version_cluster-version-operator-5c965bbfc6-9gtdt_fde7973f-1003-4839-a94c-91f713c5719b/cluster-version-operator/ 2026-03-20 16:36:41.683359 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-cluster-version_cluster-version-operator-5c965bbfc6-9gtdt_fde7973f-1003-4839-a94c-91f713c5719b/cluster-version-operator/0.log.gz 2026-03-20 16:36:41.683383 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-config-operator_openshift-config-operator-7777fb866f-xzds4_1604f02d-e1f2-4ff6-8964-e038f8a64864/ 2026-03-20 16:36:41.683406 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-config-operator_openshift-config-operator-7777fb866f-xzds4_1604f02d-e1f2-4ff6-8964-e038f8a64864/openshift-api/ 2026-03-20 16:36:41.683430 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-config-operator_openshift-config-operator-7777fb866f-xzds4_1604f02d-e1f2-4ff6-8964-e038f8a64864/openshift-api/0.log 2026-03-20 16:36:41.683453 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-config-operator_openshift-config-operator-7777fb866f-xzds4_1604f02d-e1f2-4ff6-8964-e038f8a64864/openshift-config-operator/ 2026-03-20 16:36:41.683476 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-config-operator_openshift-config-operator-7777fb866f-xzds4_1604f02d-e1f2-4ff6-8964-e038f8a64864/openshift-config-operator/0.log 2026-03-20 16:36:41.683499 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-console-operator_console-operator-58897d9998-xv5bz_df346070-57b9-44a1-9e7d-6383faeea245/ 2026-03-20 16:36:41.683522 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-console-operator_console-operator-58897d9998-xv5bz_df346070-57b9-44a1-9e7d-6383faeea245/console-operator/ 2026-03-20 16:36:41.683545 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-console-operator_console-operator-58897d9998-xv5bz_df346070-57b9-44a1-9e7d-6383faeea245/console-operator/0.log 2026-03-20 16:36:41.683568 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-console_console-7b878968-6wrvq_e349ac25-222d-45bd-80c6-d2a7f74bbbdc/ 2026-03-20 16:36:41.683591 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-console_console-7b878968-6wrvq_e349ac25-222d-45bd-80c6-d2a7f74bbbdc/console/ 2026-03-20 16:36:41.683615 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-console_console-7b878968-6wrvq_e349ac25-222d-45bd-80c6-d2a7f74bbbdc/console/0.log 2026-03-20 16:36:41.683644 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-console_downloads-7954f5f757-m787j_1be7a1f1-df63-450b-aa07-3b5f76e9b6f7/ 2026-03-20 16:36:41.683689 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-console_downloads-7954f5f757-m787j_1be7a1f1-df63-450b-aa07-3b5f76e9b6f7/download-server/ 2026-03-20 16:36:41.683720 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-console_downloads-7954f5f757-m787j_1be7a1f1-df63-450b-aa07-3b5f76e9b6f7/download-server/0.log 2026-03-20 16:36:41.683745 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-controller-manager-operator_openshift-controller-manager-operator-756b6f6bc6-87msr_8a28763e-1762-4c17-8fb5-4692158a16a8/ 2026-03-20 16:36:41.683769 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-controller-manager-operator_openshift-controller-manager-operator-756b6f6bc6-87msr_8a28763e-1762-4c17-8fb5-4692158a16a8/openshift-controller-manager-operator/ 2026-03-20 16:36:41.683793 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-controller-manager-operator_openshift-controller-manager-operator-756b6f6bc6-87msr_8a28763e-1762-4c17-8fb5-4692158a16a8/openshift-controller-manager-operator/0.log 2026-03-20 16:36:41.683817 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-controller-manager_controller-manager-59d77b64d6-8rzs8_264536e9-9ded-4743-a26e-24cd5a55e2ac/ 2026-03-20 16:36:41.683841 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-controller-manager_controller-manager-59d77b64d6-8rzs8_264536e9-9ded-4743-a26e-24cd5a55e2ac/controller-manager/ 2026-03-20 16:36:41.683864 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-controller-manager_controller-manager-59d77b64d6-8rzs8_264536e9-9ded-4743-a26e-24cd5a55e2ac/controller-manager/0.log 2026-03-20 16:36:41.683889 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns-operator_dns-operator-744455d44c-b6pj5_8f783405-c032-4733-8e15-0dce88717000/ 2026-03-20 16:36:41.683913 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns-operator_dns-operator-744455d44c-b6pj5_8f783405-c032-4733-8e15-0dce88717000/dns-operator/ 2026-03-20 16:36:41.684035 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns-operator_dns-operator-744455d44c-b6pj5_8f783405-c032-4733-8e15-0dce88717000/dns-operator/0.log 2026-03-20 16:36:41.684066 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns-operator_dns-operator-744455d44c-b6pj5_8f783405-c032-4733-8e15-0dce88717000/kube-rbac-proxy/ 2026-03-20 16:36:41.684091 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns-operator_dns-operator-744455d44c-b6pj5_8f783405-c032-4733-8e15-0dce88717000/kube-rbac-proxy/0.log 2026-03-20 16:36:41.684116 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns_dns-default-dnc2j_dcd2f0d3-cdac-480d-91cd-88c8a78005ba/ 2026-03-20 16:36:41.684141 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns_dns-default-dnc2j_dcd2f0d3-cdac-480d-91cd-88c8a78005ba/dns/ 2026-03-20 16:36:41.684165 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns_dns-default-dnc2j_dcd2f0d3-cdac-480d-91cd-88c8a78005ba/dns/0.log 2026-03-20 16:36:41.684189 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns_dns-default-dnc2j_dcd2f0d3-cdac-480d-91cd-88c8a78005ba/kube-rbac-proxy/ 2026-03-20 16:36:41.684212 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns_dns-default-dnc2j_dcd2f0d3-cdac-480d-91cd-88c8a78005ba/kube-rbac-proxy/0.log 2026-03-20 16:36:41.684236 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns_node-resolver-czvmm_e5dd2ff6-f564-457c-9781-eefecdfea677/ 2026-03-20 16:36:41.684269 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns_node-resolver-czvmm_e5dd2ff6-f564-457c-9781-eefecdfea677/dns-node-resolver/ 2026-03-20 16:36:41.684308 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-dns_node-resolver-czvmm_e5dd2ff6-f564-457c-9781-eefecdfea677/dns-node-resolver/0.log 2026-03-20 16:36:41.684332 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd-operator_etcd-operator-b45778765-pc9l6_b3cecb27-dbee-4cc4-be8b-79989833677e/ 2026-03-20 16:36:41.684356 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd-operator_etcd-operator-b45778765-pc9l6_b3cecb27-dbee-4cc4-be8b-79989833677e/etcd-operator/ 2026-03-20 16:36:41.684379 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd-operator_etcd-operator-b45778765-pc9l6_b3cecb27-dbee-4cc4-be8b-79989833677e/etcd-operator/0.log 2026-03-20 16:36:41.684402 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/ 2026-03-20 16:36:41.684425 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-ensure-env-vars/ 2026-03-20 16:36:41.684448 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-ensure-env-vars/0.log 2026-03-20 16:36:41.684471 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-metrics/ 2026-03-20 16:36:41.684494 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-metrics/0.log 2026-03-20 16:36:41.684517 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-readyz/ 2026-03-20 16:36:41.684539 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-readyz/0.log 2026-03-20 16:36:41.684562 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-resources-copy/ 2026-03-20 16:36:41.684584 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-resources-copy/0.log 2026-03-20 16:36:41.684607 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-rev/ 2026-03-20 16:36:41.684630 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd-rev/0.log 2026-03-20 16:36:41.684652 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd/ 2026-03-20 16:36:41.684709 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcd/0.log 2026-03-20 16:36:41.684736 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcdctl/ 2026-03-20 16:36:41.684760 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/etcdctl/0.log 2026-03-20 16:36:41.684783 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/setup/ 2026-03-20 16:36:41.684807 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-etcd_etcd-crc_2139d3e2895fc6797b9c76a1b4c9886d/setup/0.log 2026-03-20 16:36:41.684835 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-image-registry_cluster-image-registry-operator-dc59b4c8b-6rhtw_b76d626e-d614-41c8-bcf3-3df8fc57b668/ 2026-03-20 16:36:41.684859 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-image-registry_cluster-image-registry-operator-dc59b4c8b-6rhtw_b76d626e-d614-41c8-bcf3-3df8fc57b668/cluster-image-registry-operator/ 2026-03-20 16:36:41.684882 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-image-registry_cluster-image-registry-operator-dc59b4c8b-6rhtw_b76d626e-d614-41c8-bcf3-3df8fc57b668/cluster-image-registry-operator/0.log 2026-03-20 16:36:41.684906 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-image-registry_image-registry-66df7c8f76-qb57x_704968cf-4e29-45ce-9b8b-610920e4a5b1/ 2026-03-20 16:36:41.684930 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-image-registry_image-registry-66df7c8f76-qb57x_704968cf-4e29-45ce-9b8b-610920e4a5b1/registry/ 2026-03-20 16:36:41.684954 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-image-registry_image-registry-66df7c8f76-qb57x_704968cf-4e29-45ce-9b8b-610920e4a5b1/registry/0.log 2026-03-20 16:36:41.684977 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-image-registry_node-ca-vx6nr_e8f0d437-97b9-4f08-8136-4a13fe545b08/ 2026-03-20 16:36:41.685000 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-image-registry_node-ca-vx6nr_e8f0d437-97b9-4f08-8136-4a13fe545b08/node-ca/ 2026-03-20 16:36:41.685023 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-image-registry_node-ca-vx6nr_e8f0d437-97b9-4f08-8136-4a13fe545b08/node-ca/0.log 2026-03-20 16:36:41.685046 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29567066-c9q56_85e8755f-951d-4604-8d18-b2c7c6d17d27/ 2026-03-20 16:36:41.685069 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29567066-c9q56_85e8755f-951d-4604-8d18-b2c7c6d17d27/oc/ 2026-03-20 16:36:41.685092 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29567066-c9q56_85e8755f-951d-4604-8d18-b2c7c6d17d27/oc/0.log 2026-03-20 16:36:41.685115 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29567068-plqw2_4537163d-0b02-40b2-ad77-ae3ec4022504/ 2026-03-20 16:36:41.685138 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29567068-plqw2_4537163d-0b02-40b2-ad77-ae3ec4022504/oc/ 2026-03-20 16:36:41.685162 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29567068-plqw2_4537163d-0b02-40b2-ad77-ae3ec4022504/oc/0.log 2026-03-20 16:36:41.685185 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29567070-h4fjk_10c98d88-c20b-4d75-b340-5ca0096c9a0f/ 2026-03-20 16:36:41.685208 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29567070-h4fjk_10c98d88-c20b-4d75-b340-5ca0096c9a0f/oc/ 2026-03-20 16:36:41.685231 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29567070-h4fjk_10c98d88-c20b-4d75-b340-5ca0096c9a0f/oc/0.log 2026-03-20 16:36:41.685254 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29567072-j9lds_b9dfcdbd-594f-46c7-96f5-9ee6decc42d7/ 2026-03-20 16:36:41.685277 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29567072-j9lds_b9dfcdbd-594f-46c7-96f5-9ee6decc42d7/oc/ 2026-03-20 16:36:41.685306 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-infra_auto-csr-approver-29567072-j9lds_b9dfcdbd-594f-46c7-96f5-9ee6decc42d7/oc/0.log 2026-03-20 16:36:41.685329 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress-canary_ingress-canary-vvcfm_3ac360b6-7201-4b96-8aa8-4e6a54d9a918/ 2026-03-20 16:36:41.685353 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress-canary_ingress-canary-vvcfm_3ac360b6-7201-4b96-8aa8-4e6a54d9a918/serve-healthcheck-canary/ 2026-03-20 16:36:41.685376 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress-canary_ingress-canary-vvcfm_3ac360b6-7201-4b96-8aa8-4e6a54d9a918/serve-healthcheck-canary/0.log 2026-03-20 16:36:41.685399 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress-operator_ingress-operator-5b745b69d9-d4pzs_df489f65-0ef7-4fd8-a402-9e816dd5620d/ 2026-03-20 16:36:41.685421 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress-operator_ingress-operator-5b745b69d9-d4pzs_df489f65-0ef7-4fd8-a402-9e816dd5620d/ingress-operator/ 2026-03-20 16:36:41.685444 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress-operator_ingress-operator-5b745b69d9-d4pzs_df489f65-0ef7-4fd8-a402-9e816dd5620d/ingress-operator/0.log 2026-03-20 16:36:41.685466 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress-operator_ingress-operator-5b745b69d9-d4pzs_df489f65-0ef7-4fd8-a402-9e816dd5620d/kube-rbac-proxy/ 2026-03-20 16:36:41.685489 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress-operator_ingress-operator-5b745b69d9-d4pzs_df489f65-0ef7-4fd8-a402-9e816dd5620d/kube-rbac-proxy/0.log 2026-03-20 16:36:41.685511 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress_router-default-5444994796-nr4cq_3ab628f6-a5aa-4cf7-af1e-774587397924/ 2026-03-20 16:36:41.685534 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress_router-default-5444994796-nr4cq_3ab628f6-a5aa-4cf7-af1e-774587397924/router/ 2026-03-20 16:36:41.685556 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ingress_router-default-5444994796-nr4cq_3ab628f6-a5aa-4cf7-af1e-774587397924/router/0.log 2026-03-20 16:36:41.685579 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver-operator_kube-apiserver-operator-766d6c64bb-5qnvb_482878eb-18a6-4c17-9d27-fc9a5fb650f2/ 2026-03-20 16:36:41.685602 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver-operator_kube-apiserver-operator-766d6c64bb-5qnvb_482878eb-18a6-4c17-9d27-fc9a5fb650f2/kube-apiserver-operator/ 2026-03-20 16:36:41.685625 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver-operator_kube-apiserver-operator-766d6c64bb-5qnvb_482878eb-18a6-4c17-9d27-fc9a5fb650f2/kube-apiserver-operator/0.log 2026-03-20 16:36:41.685648 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_installer-9-crc_e5b57c2d-92a5-479c-bcb3-5382a04091fc/ 2026-03-20 16:36:41.685696 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_installer-9-crc_e5b57c2d-92a5-479c-bcb3-5382a04091fc/installer/ 2026-03-20 16:36:41.685722 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_installer-9-crc_e5b57c2d-92a5-479c-bcb3-5382a04091fc/installer/0.log 2026-03-20 16:36:41.685746 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/ 2026-03-20 16:36:41.685768 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver-cert-regeneration-controller/ 2026-03-20 16:36:41.685797 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver-cert-regeneration-controller/0.log 2026-03-20 16:36:41.685821 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver-cert-syncer/ 2026-03-20 16:36:41.685851 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver-cert-syncer/0.log 2026-03-20 16:36:41.685880 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver-check-endpoints/ 2026-03-20 16:36:41.685957 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver-check-endpoints/0.log 2026-03-20 16:36:41.685985 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver-insecure-readyz/ 2026-03-20 16:36:41.686009 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver-insecure-readyz/0.log 2026-03-20 16:36:41.686032 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver/ 2026-03-20 16:36:41.686055 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/kube-apiserver/0.log 2026-03-20 16:36:41.686079 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/setup/ 2026-03-20 16:36:41.686101 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_kube-apiserver-crc_71bb4a3aecc4ba5b26c4b7318770ce13/setup/0.log 2026-03-20 16:36:41.686124 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_revision-pruner-8-crc_d2241316-a31e-4144-80db-7e75c432c8d1/ 2026-03-20 16:36:41.686146 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_revision-pruner-8-crc_d2241316-a31e-4144-80db-7e75c432c8d1/pruner/ 2026-03-20 16:36:41.686169 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_revision-pruner-8-crc_d2241316-a31e-4144-80db-7e75c432c8d1/pruner/0.log 2026-03-20 16:36:41.686192 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_revision-pruner-9-crc_9861ea25-b65a-41f1-940d-b0635cd8cf67/ 2026-03-20 16:36:41.686215 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_revision-pruner-9-crc_9861ea25-b65a-41f1-940d-b0635cd8cf67/pruner/ 2026-03-20 16:36:41.686239 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-apiserver_revision-pruner-9-crc_9861ea25-b65a-41f1-940d-b0635cd8cf67/pruner/0.log 2026-03-20 16:36:41.686262 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager-operator_kube-controller-manager-operator-78b949d7b-m6czc_2f466308-78b7-43ba-bbaa-8de7afa3d22b/ 2026-03-20 16:36:41.686286 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager-operator_kube-controller-manager-operator-78b949d7b-m6czc_2f466308-78b7-43ba-bbaa-8de7afa3d22b/kube-controller-manager-operator/ 2026-03-20 16:36:41.686315 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager-operator_kube-controller-manager-operator-78b949d7b-m6czc_2f466308-78b7-43ba-bbaa-8de7afa3d22b/kube-controller-manager-operator/0.log 2026-03-20 16:36:41.686340 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/ 2026-03-20 16:36:41.686363 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/cluster-policy-controller/ 2026-03-20 16:36:41.686387 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/cluster-policy-controller/1.log 2026-03-20 16:36:41.686411 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/cluster-policy-controller/2.log 2026-03-20 16:36:41.686434 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/kube-controller-manager-cert-syncer/ 2026-03-20 16:36:41.686458 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/kube-controller-manager-cert-syncer/0.log 2026-03-20 16:36:41.686481 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/kube-controller-manager-recovery-controller/ 2026-03-20 16:36:41.686505 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/kube-controller-manager-recovery-controller/0.log 2026-03-20 16:36:41.686528 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/kube-controller-manager/ 2026-03-20 16:36:41.686551 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/kube-controller-manager/0.log 2026-03-20 16:36:41.686574 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_kube-controller-manager-crc_f614b9022728cf315e60c057852e563e/kube-controller-manager/1.log.gz 2026-03-20 16:36:41.686597 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_revision-pruner-9-crc_2d8518e7-60ec-45a4-96da-b1cea470296a/ 2026-03-20 16:36:41.686620 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_revision-pruner-9-crc_2d8518e7-60ec-45a4-96da-b1cea470296a/pruner/ 2026-03-20 16:36:41.686643 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-controller-manager_revision-pruner-9-crc_2d8518e7-60ec-45a4-96da-b1cea470296a/pruner/0.log 2026-03-20 16:36:41.686711 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler-operator_openshift-kube-scheduler-operator-5fdd9b5758-wqkfl_3aecd414-5e15-473d-af30-c967633f216c/ 2026-03-20 16:36:41.686743 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler-operator_openshift-kube-scheduler-operator-5fdd9b5758-wqkfl_3aecd414-5e15-473d-af30-c967633f216c/kube-scheduler-operator-container/ 2026-03-20 16:36:41.686768 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler-operator_openshift-kube-scheduler-operator-5fdd9b5758-wqkfl_3aecd414-5e15-473d-af30-c967633f216c/kube-scheduler-operator-container/0.log 2026-03-20 16:36:41.686796 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler_openshift-kube-scheduler-crc_3dcd261975c3d6b9a6ad6367fd4facd3/ 2026-03-20 16:36:41.686820 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler_openshift-kube-scheduler-crc_3dcd261975c3d6b9a6ad6367fd4facd3/kube-scheduler-cert-syncer/ 2026-03-20 16:36:41.686844 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler_openshift-kube-scheduler-crc_3dcd261975c3d6b9a6ad6367fd4facd3/kube-scheduler-cert-syncer/0.log 2026-03-20 16:36:41.686867 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler_openshift-kube-scheduler-crc_3dcd261975c3d6b9a6ad6367fd4facd3/kube-scheduler-recovery-controller/ 2026-03-20 16:36:41.686890 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler_openshift-kube-scheduler-crc_3dcd261975c3d6b9a6ad6367fd4facd3/kube-scheduler-recovery-controller/0.log 2026-03-20 16:36:41.686914 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler_openshift-kube-scheduler-crc_3dcd261975c3d6b9a6ad6367fd4facd3/kube-scheduler/ 2026-03-20 16:36:41.686937 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler_openshift-kube-scheduler-crc_3dcd261975c3d6b9a6ad6367fd4facd3/kube-scheduler/0.log 2026-03-20 16:36:41.686960 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler_openshift-kube-scheduler-crc_3dcd261975c3d6b9a6ad6367fd4facd3/wait-for-host-port/ 2026-03-20 16:36:41.686984 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-scheduler_openshift-kube-scheduler-crc_3dcd261975c3d6b9a6ad6367fd4facd3/wait-for-host-port/0.log 2026-03-20 16:36:41.687007 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-b67b599dd-bfhd6_e6ff8412-c6e3-43a6-92e3-1644d33ee12f/ 2026-03-20 16:36:41.687031 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-b67b599dd-bfhd6_e6ff8412-c6e3-43a6-92e3-1644d33ee12f/kube-storage-version-migrator-operator/ 2026-03-20 16:36:41.687055 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-b67b599dd-bfhd6_e6ff8412-c6e3-43a6-92e3-1644d33ee12f/kube-storage-version-migrator-operator/0.log 2026-03-20 16:36:41.687079 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-storage-version-migrator_migrator-59844c95c7-8h9gl_3b3ed106-44fd-4a77-a80e-d5e4faa7350a/ 2026-03-20 16:36:41.687103 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-storage-version-migrator_migrator-59844c95c7-8h9gl_3b3ed106-44fd-4a77-a80e-d5e4faa7350a/graceful-termination/ 2026-03-20 16:36:41.687127 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-storage-version-migrator_migrator-59844c95c7-8h9gl_3b3ed106-44fd-4a77-a80e-d5e4faa7350a/graceful-termination/0.log 2026-03-20 16:36:41.687150 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-storage-version-migrator_migrator-59844c95c7-8h9gl_3b3ed106-44fd-4a77-a80e-d5e4faa7350a/migrator/ 2026-03-20 16:36:41.687173 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-kube-storage-version-migrator_migrator-59844c95c7-8h9gl_3b3ed106-44fd-4a77-a80e-d5e4faa7350a/migrator/0.log 2026-03-20 16:36:41.687196 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-api_control-plane-machine-set-operator-78cbb6b69f-lzrhk_1dce783b-8d1e-4171-8409-be2d773e2ab0/ 2026-03-20 16:36:41.687227 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-api_control-plane-machine-set-operator-78cbb6b69f-lzrhk_1dce783b-8d1e-4171-8409-be2d773e2ab0/control-plane-machine-set-operator/ 2026-03-20 16:36:41.687251 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-api_control-plane-machine-set-operator-78cbb6b69f-lzrhk_1dce783b-8d1e-4171-8409-be2d773e2ab0/control-plane-machine-set-operator/0.log 2026-03-20 16:36:41.687275 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-api_machine-api-operator-5694c8668f-hszjg_ef59e32b-dd5c-4beb-b348-67c4847e80ce/ 2026-03-20 16:36:41.687298 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-api_machine-api-operator-5694c8668f-hszjg_ef59e32b-dd5c-4beb-b348-67c4847e80ce/kube-rbac-proxy/ 2026-03-20 16:36:41.687321 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-api_machine-api-operator-5694c8668f-hszjg_ef59e32b-dd5c-4beb-b348-67c4847e80ce/kube-rbac-proxy/0.log 2026-03-20 16:36:41.687344 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-api_machine-api-operator-5694c8668f-hszjg_ef59e32b-dd5c-4beb-b348-67c4847e80ce/machine-api-operator/ 2026-03-20 16:36:41.687373 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-api_machine-api-operator-5694c8668f-hszjg_ef59e32b-dd5c-4beb-b348-67c4847e80ce/machine-api-operator/0.log 2026-03-20 16:36:41.687397 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-crc_d1b160f5dda77d281dd8e69ec8d817f9/ 2026-03-20 16:36:41.687421 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-crc_d1b160f5dda77d281dd8e69ec8d817f9/kube-rbac-proxy-crio/ 2026-03-20 16:36:41.687444 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-crc_d1b160f5dda77d281dd8e69ec8d817f9/kube-rbac-proxy-crio/0.log 2026-03-20 16:36:41.687467 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-crc_d1b160f5dda77d281dd8e69ec8d817f9/setup/ 2026-03-20 16:36:41.687490 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-crc_d1b160f5dda77d281dd8e69ec8d817f9/setup/0.log 2026-03-20 16:36:41.687514 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-controller-84d6567774-62nz7_57db9fef-f17d-4d96-b830-8d79612fdfde/ 2026-03-20 16:36:41.687537 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-controller-84d6567774-62nz7_57db9fef-f17d-4d96-b830-8d79612fdfde/kube-rbac-proxy/ 2026-03-20 16:36:41.687560 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-controller-84d6567774-62nz7_57db9fef-f17d-4d96-b830-8d79612fdfde/kube-rbac-proxy/0.log 2026-03-20 16:36:41.687583 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-controller-84d6567774-62nz7_57db9fef-f17d-4d96-b830-8d79612fdfde/machine-config-controller/ 2026-03-20 16:36:41.687607 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-controller-84d6567774-62nz7_57db9fef-f17d-4d96-b830-8d79612fdfde/machine-config-controller/0.log 2026-03-20 16:36:41.687630 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-daemon-l8d6t_dbc04883-b38a-4b6a-bee4-f6804c8aad94/ 2026-03-20 16:36:41.687658 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-daemon-l8d6t_dbc04883-b38a-4b6a-bee4-f6804c8aad94/kube-rbac-proxy/ 2026-03-20 16:36:41.687709 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-daemon-l8d6t_dbc04883-b38a-4b6a-bee4-f6804c8aad94/kube-rbac-proxy/0.log 2026-03-20 16:36:41.687734 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-daemon-l8d6t_dbc04883-b38a-4b6a-bee4-f6804c8aad94/machine-config-daemon/ 2026-03-20 16:36:41.687758 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-daemon-l8d6t_dbc04883-b38a-4b6a-bee4-f6804c8aad94/machine-config-daemon/11.log 2026-03-20 16:36:41.687781 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-operator-74547568cd-pvn9m_aae73cf2-faff-476e-8888-450853cea687/ 2026-03-20 16:36:41.687805 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-operator-74547568cd-pvn9m_aae73cf2-faff-476e-8888-450853cea687/kube-rbac-proxy/ 2026-03-20 16:36:41.687828 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-operator-74547568cd-pvn9m_aae73cf2-faff-476e-8888-450853cea687/kube-rbac-proxy/0.log 2026-03-20 16:36:41.687852 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-operator-74547568cd-pvn9m_aae73cf2-faff-476e-8888-450853cea687/machine-config-operator/ 2026-03-20 16:36:41.687875 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-operator-74547568cd-pvn9m_aae73cf2-faff-476e-8888-450853cea687/machine-config-operator/0.log 2026-03-20 16:36:41.687899 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-server-r65dl_2f673f37-d358-4733-b2f2-3a7996b7615b/ 2026-03-20 16:36:41.687922 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-server-r65dl_2f673f37-d358-4733-b2f2-3a7996b7615b/machine-config-server/ 2026-03-20 16:36:41.687945 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-machine-config-operator_machine-config-server-r65dl_2f673f37-d358-4733-b2f2-3a7996b7615b/machine-config-server/0.log 2026-03-20 16:36:41.687969 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde874crv8c_0259c0a9-34c2-4000-982b-ae1122f345fc/ 2026-03-20 16:36:41.687992 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde874crv8c_0259c0a9-34c2-4000-982b-ae1122f345fc/extract/ 2026-03-20 16:36:41.688016 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde874crv8c_0259c0a9-34c2-4000-982b-ae1122f345fc/extract/0.log 2026-03-20 16:36:41.688039 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde874crv8c_0259c0a9-34c2-4000-982b-ae1122f345fc/pull/ 2026-03-20 16:36:41.688062 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde874crv8c_0259c0a9-34c2-4000-982b-ae1122f345fc/pull/0.log 2026-03-20 16:36:41.688085 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde874crv8c_0259c0a9-34c2-4000-982b-ae1122f345fc/util/ 2026-03-20 16:36:41.688114 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde874crv8c_0259c0a9-34c2-4000-982b-ae1122f345fc/util/0.log 2026-03-20 16:36:41.688138 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c167lkk_60e1c3ce-b711-4bac-afd3-60804a46154f/ 2026-03-20 16:36:41.688162 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c167lkk_60e1c3ce-b711-4bac-afd3-60804a46154f/extract/ 2026-03-20 16:36:41.688185 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c167lkk_60e1c3ce-b711-4bac-afd3-60804a46154f/extract/0.log 2026-03-20 16:36:41.688209 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c167lkk_60e1c3ce-b711-4bac-afd3-60804a46154f/pull/ 2026-03-20 16:36:41.688232 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c167lkk_60e1c3ce-b711-4bac-afd3-60804a46154f/pull/0.log 2026-03-20 16:36:41.688255 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c167lkk_60e1c3ce-b711-4bac-afd3-60804a46154f/util/ 2026-03-20 16:36:41.688279 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c167lkk_60e1c3ce-b711-4bac-afd3-60804a46154f/util/0.log 2026-03-20 16:36:41.688317 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5vx5n9_665d8eab-08d7-4b69-b20c-75c8151bb3b5/ 2026-03-20 16:36:41.688342 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5vx5n9_665d8eab-08d7-4b69-b20c-75c8151bb3b5/extract/ 2026-03-20 16:36:41.688366 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5vx5n9_665d8eab-08d7-4b69-b20c-75c8151bb3b5/extract/0.log 2026-03-20 16:36:41.688390 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5vx5n9_665d8eab-08d7-4b69-b20c-75c8151bb3b5/pull/ 2026-03-20 16:36:41.688413 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5vx5n9_665d8eab-08d7-4b69-b20c-75c8151bb3b5/pull/0.log 2026-03-20 16:36:41.688437 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5vx5n9_665d8eab-08d7-4b69-b20c-75c8151bb3b5/util/ 2026-03-20 16:36:41.688460 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5vx5n9_665d8eab-08d7-4b69-b20c-75c8151bb3b5/util/0.log 2026-03-20 16:36:41.688483 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_93d662022be5376a0ed3676a120a68427f47e4653a19a985adf923972657c7m_ee83deb1-a281-4eb3-9ba2-7212ae713ae6/ 2026-03-20 16:36:41.688506 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_93d662022be5376a0ed3676a120a68427f47e4653a19a985adf923972657c7m_ee83deb1-a281-4eb3-9ba2-7212ae713ae6/extract/ 2026-03-20 16:36:41.688535 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_93d662022be5376a0ed3676a120a68427f47e4653a19a985adf923972657c7m_ee83deb1-a281-4eb3-9ba2-7212ae713ae6/extract/0.log 2026-03-20 16:36:41.688559 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_93d662022be5376a0ed3676a120a68427f47e4653a19a985adf923972657c7m_ee83deb1-a281-4eb3-9ba2-7212ae713ae6/pull/ 2026-03-20 16:36:41.688583 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_93d662022be5376a0ed3676a120a68427f47e4653a19a985adf923972657c7m_ee83deb1-a281-4eb3-9ba2-7212ae713ae6/pull/0.log 2026-03-20 16:36:41.688606 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_93d662022be5376a0ed3676a120a68427f47e4653a19a985adf923972657c7m_ee83deb1-a281-4eb3-9ba2-7212ae713ae6/util/ 2026-03-20 16:36:41.688630 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_93d662022be5376a0ed3676a120a68427f47e4653a19a985adf923972657c7m_ee83deb1-a281-4eb3-9ba2-7212ae713ae6/util/0.log 2026-03-20 16:36:41.688652 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_certified-operators-bfbr2_e0d2a4c2-86dc-48d5-90e0-3fcc872046b6/ 2026-03-20 16:36:41.688702 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_certified-operators-bfbr2_e0d2a4c2-86dc-48d5-90e0-3fcc872046b6/extract-content/ 2026-03-20 16:36:41.688728 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_certified-operators-bfbr2_e0d2a4c2-86dc-48d5-90e0-3fcc872046b6/extract-content/0.log 2026-03-20 16:36:41.688752 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_certified-operators-bfbr2_e0d2a4c2-86dc-48d5-90e0-3fcc872046b6/extract-utilities/ 2026-03-20 16:36:41.688775 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_certified-operators-bfbr2_e0d2a4c2-86dc-48d5-90e0-3fcc872046b6/extract-utilities/0.log 2026-03-20 16:36:41.688798 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_certified-operators-bfbr2_e0d2a4c2-86dc-48d5-90e0-3fcc872046b6/registry-server/ 2026-03-20 16:36:41.688821 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_certified-operators-bfbr2_e0d2a4c2-86dc-48d5-90e0-3fcc872046b6/registry-server/0.log.gz 2026-03-20 16:36:41.688845 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_community-operators-bwvk2_72d4af38-f89a-48ae-bd87-96c7d479310e/ 2026-03-20 16:36:41.688867 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_community-operators-bwvk2_72d4af38-f89a-48ae-bd87-96c7d479310e/extract-content/ 2026-03-20 16:36:41.688890 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_community-operators-bwvk2_72d4af38-f89a-48ae-bd87-96c7d479310e/extract-content/0.log 2026-03-20 16:36:41.688913 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_community-operators-bwvk2_72d4af38-f89a-48ae-bd87-96c7d479310e/extract-utilities/ 2026-03-20 16:36:41.688935 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_community-operators-bwvk2_72d4af38-f89a-48ae-bd87-96c7d479310e/extract-utilities/0.log 2026-03-20 16:36:41.688958 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_community-operators-bwvk2_72d4af38-f89a-48ae-bd87-96c7d479310e/registry-server/ 2026-03-20 16:36:41.688981 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_community-operators-bwvk2_72d4af38-f89a-48ae-bd87-96c7d479310e/registry-server/0.log.gz 2026-03-20 16:36:41.689009 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_marketplace-operator-79b997595-qrqgm_d700a123-9a6b-4d44-89b2-73e09fe026b3/ 2026-03-20 16:36:41.689032 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_marketplace-operator-79b997595-qrqgm_d700a123-9a6b-4d44-89b2-73e09fe026b3/marketplace-operator/ 2026-03-20 16:36:41.689055 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_marketplace-operator-79b997595-qrqgm_d700a123-9a6b-4d44-89b2-73e09fe026b3/marketplace-operator/0.log 2026-03-20 16:36:41.689077 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-marketplace-22jh6_d857b935-7153-4ddb-bec2-b55fff235cd4/ 2026-03-20 16:36:41.689100 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-marketplace-22jh6_d857b935-7153-4ddb-bec2-b55fff235cd4/extract-content/ 2026-03-20 16:36:41.689123 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-marketplace-22jh6_d857b935-7153-4ddb-bec2-b55fff235cd4/extract-content/0.log 2026-03-20 16:36:41.689146 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-marketplace-22jh6_d857b935-7153-4ddb-bec2-b55fff235cd4/extract-utilities/ 2026-03-20 16:36:41.689170 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-marketplace-22jh6_d857b935-7153-4ddb-bec2-b55fff235cd4/extract-utilities/0.log 2026-03-20 16:36:41.689193 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-marketplace-22jh6_d857b935-7153-4ddb-bec2-b55fff235cd4/registry-server/ 2026-03-20 16:36:41.689216 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-marketplace-22jh6_d857b935-7153-4ddb-bec2-b55fff235cd4/registry-server/0.log 2026-03-20 16:36:41.689239 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-operators-vpvn4_77c0d32a-5038-4f35-9faa-8e9cf832b213/ 2026-03-20 16:36:41.689262 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-operators-vpvn4_77c0d32a-5038-4f35-9faa-8e9cf832b213/extract-content/ 2026-03-20 16:36:41.689286 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-operators-vpvn4_77c0d32a-5038-4f35-9faa-8e9cf832b213/extract-content/0.log 2026-03-20 16:36:41.689310 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-operators-vpvn4_77c0d32a-5038-4f35-9faa-8e9cf832b213/extract-utilities/ 2026-03-20 16:36:41.689332 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-operators-vpvn4_77c0d32a-5038-4f35-9faa-8e9cf832b213/extract-utilities/0.log 2026-03-20 16:36:41.689355 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-operators-vpvn4_77c0d32a-5038-4f35-9faa-8e9cf832b213/registry-server/ 2026-03-20 16:36:41.689378 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-marketplace_redhat-operators-vpvn4_77c0d32a-5038-4f35-9faa-8e9cf832b213/registry-server/0.log.gz 2026-03-20 16:36:41.689401 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-rsfpf_f0207723-5d01-4034-bdcd-5bb28ff71021/ 2026-03-20 16:36:41.689424 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-rsfpf_f0207723-5d01-4034-bdcd-5bb28ff71021/bond-cni-plugin/ 2026-03-20 16:36:41.689447 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-rsfpf_f0207723-5d01-4034-bdcd-5bb28ff71021/bond-cni-plugin/0.log 2026-03-20 16:36:41.689476 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-rsfpf_f0207723-5d01-4034-bdcd-5bb28ff71021/cni-plugins/ 2026-03-20 16:36:41.689547 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-rsfpf_f0207723-5d01-4034-bdcd-5bb28ff71021/cni-plugins/0.log 2026-03-20 16:36:41.689572 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-rsfpf_f0207723-5d01-4034-bdcd-5bb28ff71021/egress-router-binary-copy/ 2026-03-20 16:36:41.689595 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-rsfpf_f0207723-5d01-4034-bdcd-5bb28ff71021/egress-router-binary-copy/0.log 2026-03-20 16:36:41.689618 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-rsfpf_f0207723-5d01-4034-bdcd-5bb28ff71021/kube-multus-additional-cni-plugins/ 2026-03-20 16:36:41.689640 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-rsfpf_f0207723-5d01-4034-bdcd-5bb28ff71021/kube-multus-additional-cni-plugins/0.log 2026-03-20 16:36:41.689683 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-rsfpf_f0207723-5d01-4034-bdcd-5bb28ff71021/routeoverride-cni/ 2026-03-20 16:36:41.689714 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-rsfpf_f0207723-5d01-4034-bdcd-5bb28ff71021/routeoverride-cni/0.log 2026-03-20 16:36:41.689738 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-rsfpf_f0207723-5d01-4034-bdcd-5bb28ff71021/whereabouts-cni-bincopy/ 2026-03-20 16:36:41.689762 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-rsfpf_f0207723-5d01-4034-bdcd-5bb28ff71021/whereabouts-cni-bincopy/0.log 2026-03-20 16:36:41.689785 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-rsfpf_f0207723-5d01-4034-bdcd-5bb28ff71021/whereabouts-cni/ 2026-03-20 16:36:41.689808 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-additional-cni-plugins-rsfpf_f0207723-5d01-4034-bdcd-5bb28ff71021/whereabouts-cni/0.log 2026-03-20 16:36:41.689832 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-admission-controller-857f4d67dd-62k4m_e6e86644-c67e-459b-9ad7-9f58409329a8/ 2026-03-20 16:36:41.689855 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-admission-controller-857f4d67dd-62k4m_e6e86644-c67e-459b-9ad7-9f58409329a8/kube-rbac-proxy/ 2026-03-20 16:36:41.689878 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-admission-controller-857f4d67dd-62k4m_e6e86644-c67e-459b-9ad7-9f58409329a8/kube-rbac-proxy/0.log 2026-03-20 16:36:41.689902 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-admission-controller-857f4d67dd-62k4m_e6e86644-c67e-459b-9ad7-9f58409329a8/multus-admission-controller/ 2026-03-20 16:36:41.689926 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-admission-controller-857f4d67dd-62k4m_e6e86644-c67e-459b-9ad7-9f58409329a8/multus-admission-controller/0.log 2026-03-20 16:36:41.689949 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-gpmgw_a76927f1-3a5b-4e2d-b1c2-b1cb3759b1c1/ 2026-03-20 16:36:41.689979 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-gpmgw_a76927f1-3a5b-4e2d-b1c2-b1cb3759b1c1/kube-multus/ 2026-03-20 16:36:41.690002 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-gpmgw_a76927f1-3a5b-4e2d-b1c2-b1cb3759b1c1/kube-multus/2.log 2026-03-20 16:36:41.690026 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_multus-gpmgw_a76927f1-3a5b-4e2d-b1c2-b1cb3759b1c1/kube-multus/3.log 2026-03-20 16:36:41.690049 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_network-metrics-daemon-lc5px_5e5f925a-75e4-485f-9d5e-2be4c2c13616/ 2026-03-20 16:36:41.690071 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_network-metrics-daemon-lc5px_5e5f925a-75e4-485f-9d5e-2be4c2c13616/kube-rbac-proxy/ 2026-03-20 16:36:41.690094 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_network-metrics-daemon-lc5px_5e5f925a-75e4-485f-9d5e-2be4c2c13616/kube-rbac-proxy/0.log 2026-03-20 16:36:41.690117 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_network-metrics-daemon-lc5px_5e5f925a-75e4-485f-9d5e-2be4c2c13616/network-metrics-daemon/ 2026-03-20 16:36:41.690139 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-multus_network-metrics-daemon-lc5px_5e5f925a-75e4-485f-9d5e-2be4c2c13616/network-metrics-daemon/0.log 2026-03-20 16:36:41.690162 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-must-gather-9gf9c_must-gather-jf28n_83b66982-c445-4603-9068-896e7c713661/ 2026-03-20 16:36:41.690185 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-must-gather-9gf9c_must-gather-jf28n_83b66982-c445-4603-9068-896e7c713661/copy/ 2026-03-20 16:36:41.690208 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-must-gather-9gf9c_must-gather-jf28n_83b66982-c445-4603-9068-896e7c713661/gather/ 2026-03-20 16:36:41.690231 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-console_networking-console-plugin-85b44fc459-gdk6g_5fe485a1-e14f-4c09-b5b9-f252bc42b7e8/ 2026-03-20 16:36:41.690254 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-console_networking-console-plugin-85b44fc459-gdk6g_5fe485a1-e14f-4c09-b5b9-f252bc42b7e8/networking-console-plugin/ 2026-03-20 16:36:41.690278 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-console_networking-console-plugin-85b44fc459-gdk6g_5fe485a1-e14f-4c09-b5b9-f252bc42b7e8/networking-console-plugin/0.log 2026-03-20 16:36:41.690301 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-diagnostics_network-check-source-55646444c4-trplf_9d751cbb-f2e2-430d-9754-c882a5e924a5/ 2026-03-20 16:36:41.690325 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-diagnostics_network-check-source-55646444c4-trplf_9d751cbb-f2e2-430d-9754-c882a5e924a5/check-endpoints/ 2026-03-20 16:36:41.690349 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-diagnostics_network-check-source-55646444c4-trplf_9d751cbb-f2e2-430d-9754-c882a5e924a5/check-endpoints/2.log 2026-03-20 16:36:41.690372 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-diagnostics_network-check-source-55646444c4-trplf_9d751cbb-f2e2-430d-9754-c882a5e924a5/check-endpoints/3.log 2026-03-20 16:36:41.690396 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-diagnostics_network-check-target-xd92c_3b6479f0-333b-4a96-9adf-2099afdc2447/ 2026-03-20 16:36:41.690419 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-diagnostics_network-check-target-xd92c_3b6479f0-333b-4a96-9adf-2099afdc2447/network-check-target-container/ 2026-03-20 16:36:41.690448 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-diagnostics_network-check-target-xd92c_3b6479f0-333b-4a96-9adf-2099afdc2447/network-check-target-container/0.log 2026-03-20 16:36:41.690472 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-node-identity_network-node-identity-vrzqb_ef543e1b-8068-4ea3-b32a-61027b32e95d/ 2026-03-20 16:36:41.690496 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-node-identity_network-node-identity-vrzqb_ef543e1b-8068-4ea3-b32a-61027b32e95d/approver/ 2026-03-20 16:36:41.690519 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-node-identity_network-node-identity-vrzqb_ef543e1b-8068-4ea3-b32a-61027b32e95d/approver/0.log 2026-03-20 16:36:41.690542 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-node-identity_network-node-identity-vrzqb_ef543e1b-8068-4ea3-b32a-61027b32e95d/webhook/ 2026-03-20 16:36:41.690565 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-node-identity_network-node-identity-vrzqb_ef543e1b-8068-4ea3-b32a-61027b32e95d/webhook/0.log 2026-03-20 16:36:41.690588 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-operator_iptables-alerter-4ln5h_d75a4c96-2883-4a0b-bab2-0fab2b6c0b49/ 2026-03-20 16:36:41.690611 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-operator_iptables-alerter-4ln5h_d75a4c96-2883-4a0b-bab2-0fab2b6c0b49/iptables-alerter/ 2026-03-20 16:36:41.690633 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-operator_iptables-alerter-4ln5h_d75a4c96-2883-4a0b-bab2-0fab2b6c0b49/iptables-alerter/0.log 2026-03-20 16:36:41.690656 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-operator_network-operator-58b4c7f79c-55gtf_37a5e44f-9a88-4405-be8a-b645485e7312/ 2026-03-20 16:36:41.690706 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-operator_network-operator-58b4c7f79c-55gtf_37a5e44f-9a88-4405-be8a-b645485e7312/network-operator/ 2026-03-20 16:36:41.690732 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-network-operator_network-operator-58b4c7f79c-55gtf_37a5e44f-9a88-4405-be8a-b645485e7312/network-operator/0.log 2026-03-20 16:36:41.690755 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-console-plugin-86f58fcf4-7shs2_56e21629-308a-4212-ac84-81570cb43089/ 2026-03-20 16:36:41.690779 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-console-plugin-86f58fcf4-7shs2_56e21629-308a-4212-ac84-81570cb43089/nmstate-console-plugin/ 2026-03-20 16:36:41.690802 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-console-plugin-86f58fcf4-7shs2_56e21629-308a-4212-ac84-81570cb43089/nmstate-console-plugin/0.log 2026-03-20 16:36:41.690826 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-handler-52hfr_b8bbc7f2-8378-46a1-9cc1-20c8366b22f4/ 2026-03-20 16:36:41.690849 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-handler-52hfr_b8bbc7f2-8378-46a1-9cc1-20c8366b22f4/nmstate-handler/ 2026-03-20 16:36:41.690872 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-handler-52hfr_b8bbc7f2-8378-46a1-9cc1-20c8366b22f4/nmstate-handler/0.log 2026-03-20 16:36:41.690895 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-metrics-9b8c8685d-xfs5s_474448b5-47ff-4f1b-a127-f3192ec0a283/ 2026-03-20 16:36:41.690923 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-metrics-9b8c8685d-xfs5s_474448b5-47ff-4f1b-a127-f3192ec0a283/kube-rbac-proxy/ 2026-03-20 16:36:41.690947 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-metrics-9b8c8685d-xfs5s_474448b5-47ff-4f1b-a127-f3192ec0a283/kube-rbac-proxy/0.log 2026-03-20 16:36:41.690970 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-metrics-9b8c8685d-xfs5s_474448b5-47ff-4f1b-a127-f3192ec0a283/nmstate-metrics/ 2026-03-20 16:36:41.690993 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-metrics-9b8c8685d-xfs5s_474448b5-47ff-4f1b-a127-f3192ec0a283/nmstate-metrics/0.log 2026-03-20 16:36:41.691015 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-operator-796d4cfff4-s52xz_6fd819ab-3d0b-427b-b29c-bf57d746a6a3/ 2026-03-20 16:36:41.691038 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-operator-796d4cfff4-s52xz_6fd819ab-3d0b-427b-b29c-bf57d746a6a3/nmstate-operator/ 2026-03-20 16:36:41.691061 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-operator-796d4cfff4-s52xz_6fd819ab-3d0b-427b-b29c-bf57d746a6a3/nmstate-operator/0.log 2026-03-20 16:36:41.691083 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-webhook-5f558f5558-xbp9m_980f1609-b4ad-464a-93cd-895066f63c92/ 2026-03-20 16:36:41.691105 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-webhook-5f558f5558-xbp9m_980f1609-b4ad-464a-93cd-895066f63c92/nmstate-webhook/ 2026-03-20 16:36:41.691128 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-nmstate_nmstate-webhook-5f558f5558-xbp9m_980f1609-b4ad-464a-93cd-895066f63c92/nmstate-webhook/0.log 2026-03-20 16:36:41.691151 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-oauth-apiserver_apiserver-7bbb656c7d-d7f5s_e53d954e-e40b-4f89-a4fe-82ce7ca92ddb/ 2026-03-20 16:36:41.691173 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-oauth-apiserver_apiserver-7bbb656c7d-d7f5s_e53d954e-e40b-4f89-a4fe-82ce7ca92ddb/fix-audit-permissions/ 2026-03-20 16:36:41.691195 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-oauth-apiserver_apiserver-7bbb656c7d-d7f5s_e53d954e-e40b-4f89-a4fe-82ce7ca92ddb/fix-audit-permissions/0.log 2026-03-20 16:36:41.691218 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-oauth-apiserver_apiserver-7bbb656c7d-d7f5s_e53d954e-e40b-4f89-a4fe-82ce7ca92ddb/oauth-apiserver/ 2026-03-20 16:36:41.691241 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-oauth-apiserver_apiserver-7bbb656c7d-d7f5s_e53d954e-e40b-4f89-a4fe-82ce7ca92ddb/oauth-apiserver/0.log 2026-03-20 16:36:41.691264 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_catalog-operator-68c6474976-tlhjn_fd7a9d1a-a89e-4230-b7cc-60aac33d83da/ 2026-03-20 16:36:41.691287 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_catalog-operator-68c6474976-tlhjn_fd7a9d1a-a89e-4230-b7cc-60aac33d83da/catalog-operator/ 2026-03-20 16:36:41.691311 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_catalog-operator-68c6474976-tlhjn_fd7a9d1a-a89e-4230-b7cc-60aac33d83da/catalog-operator/0.log.gz 2026-03-20 16:36:41.691334 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29567040-9bzhs_0568d2a5-e8f7-4cfd-9035-b441b7d06c17/ 2026-03-20 16:36:41.691358 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29567040-9bzhs_0568d2a5-e8f7-4cfd-9035-b441b7d06c17/collect-profiles/ 2026-03-20 16:36:41.691391 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29567040-9bzhs_0568d2a5-e8f7-4cfd-9035-b441b7d06c17/collect-profiles/0.log 2026-03-20 16:36:41.691415 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29567055-z6bf6_dea0d206-5d4e-48d7-aa3e-491be6ecf465/ 2026-03-20 16:36:41.691438 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29567055-z6bf6_dea0d206-5d4e-48d7-aa3e-491be6ecf465/collect-profiles/ 2026-03-20 16:36:41.691462 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29567055-z6bf6_dea0d206-5d4e-48d7-aa3e-491be6ecf465/collect-profiles/0.log 2026-03-20 16:36:41.691485 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29567070-lbqxx_7964d290-dc58-43ae-bbf5-461ce864849c/ 2026-03-20 16:36:41.691509 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29567070-lbqxx_7964d290-dc58-43ae-bbf5-461ce864849c/collect-profiles/ 2026-03-20 16:36:41.691533 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_collect-profiles-29567070-lbqxx_7964d290-dc58-43ae-bbf5-461ce864849c/collect-profiles/0.log 2026-03-20 16:36:41.691556 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_olm-operator-6b444d44fb-fqp6p_e1e06b90-d75c-4c56-a08d-3f71290dc764/ 2026-03-20 16:36:41.691580 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_olm-operator-6b444d44fb-fqp6p_e1e06b90-d75c-4c56-a08d-3f71290dc764/olm-operator/ 2026-03-20 16:36:41.691603 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_olm-operator-6b444d44fb-fqp6p_e1e06b90-d75c-4c56-a08d-3f71290dc764/olm-operator/0.log 2026-03-20 16:36:41.691626 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_package-server-manager-789f6589d5-5xf4w_1d44b082-02e7-4ef0-9242-cbf1b709e91b/ 2026-03-20 16:36:41.691650 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_package-server-manager-789f6589d5-5xf4w_1d44b082-02e7-4ef0-9242-cbf1b709e91b/kube-rbac-proxy/ 2026-03-20 16:36:41.691701 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_package-server-manager-789f6589d5-5xf4w_1d44b082-02e7-4ef0-9242-cbf1b709e91b/kube-rbac-proxy/0.log 2026-03-20 16:36:41.691728 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_package-server-manager-789f6589d5-5xf4w_1d44b082-02e7-4ef0-9242-cbf1b709e91b/package-server-manager/ 2026-03-20 16:36:41.691752 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_package-server-manager-789f6589d5-5xf4w_1d44b082-02e7-4ef0-9242-cbf1b709e91b/package-server-manager/0.log 2026-03-20 16:36:41.691775 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_packageserver-d55dfcdfc-dlbqb_71b1bc2c-4da8-49e1-b3d4-d27901ea92bf/ 2026-03-20 16:36:41.691798 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_packageserver-d55dfcdfc-dlbqb_71b1bc2c-4da8-49e1-b3d4-d27901ea92bf/packageserver/ 2026-03-20 16:36:41.691821 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operator-lifecycle-manager_packageserver-d55dfcdfc-dlbqb_71b1bc2c-4da8-49e1-b3d4-d27901ea92bf/packageserver/0.log 2026-03-20 16:36:41.691850 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_obo-prometheus-operator-8ff7d675-zbjnw_13a0d7fe-7887-4ea4-ae5b-b47d8689373b/ 2026-03-20 16:36:41.691873 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_obo-prometheus-operator-8ff7d675-zbjnw_13a0d7fe-7887-4ea4-ae5b-b47d8689373b/prometheus-operator/ 2026-03-20 16:36:41.691897 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_obo-prometheus-operator-8ff7d675-zbjnw_13a0d7fe-7887-4ea4-ae5b-b47d8689373b/prometheus-operator/0.log 2026-03-20 16:36:41.691921 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_obo-prometheus-operator-admission-webhook-6b7d587c88-7c8zn_fb3c3550-7cd9-4286-8ab3-e554c159b357/ 2026-03-20 16:36:41.691945 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_obo-prometheus-operator-admission-webhook-6b7d587c88-7c8zn_fb3c3550-7cd9-4286-8ab3-e554c159b357/prometheus-operator-admission-webhook/ 2026-03-20 16:36:41.691969 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_obo-prometheus-operator-admission-webhook-6b7d587c88-7c8zn_fb3c3550-7cd9-4286-8ab3-e554c159b357/prometheus-operator-admission-webhook/0.log 2026-03-20 16:36:41.691993 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_obo-prometheus-operator-admission-webhook-6b7d587c88-vcgbb_4ac554a5-8e12-415d-845c-72c909a7d1d2/ 2026-03-20 16:36:41.692017 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_obo-prometheus-operator-admission-webhook-6b7d587c88-vcgbb_4ac554a5-8e12-415d-845c-72c909a7d1d2/prometheus-operator-admission-webhook/ 2026-03-20 16:36:41.692040 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_obo-prometheus-operator-admission-webhook-6b7d587c88-vcgbb_4ac554a5-8e12-415d-845c-72c909a7d1d2/prometheus-operator-admission-webhook/0.log 2026-03-20 16:36:41.692063 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_observability-operator-6dd7dd855f-bvk5s_9cbfb8c9-4eb6-4382-b8b1-572029001cc0/ 2026-03-20 16:36:41.692086 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_observability-operator-6dd7dd855f-bvk5s_9cbfb8c9-4eb6-4382-b8b1-572029001cc0/operator/ 2026-03-20 16:36:41.692109 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_observability-operator-6dd7dd855f-bvk5s_9cbfb8c9-4eb6-4382-b8b1-572029001cc0/operator/0.log 2026-03-20 16:36:41.692132 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_observability-ui-dashboards-7f87b9b85b-6t9mn_6c22deaf-5054-462f-bc80-f21fedd71c41/ 2026-03-20 16:36:41.692155 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_observability-ui-dashboards-7f87b9b85b-6t9mn_6c22deaf-5054-462f-bc80-f21fedd71c41/observability-ui-dashboards/ 2026-03-20 16:36:41.692178 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_observability-ui-dashboards-7f87b9b85b-6t9mn_6c22deaf-5054-462f-bc80-f21fedd71c41/observability-ui-dashboards/0.log 2026-03-20 16:36:41.692201 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_perses-operator-b9bc87685-dshn8_95615498-4862-4d52-9c91-6f2f4f17d4ba/ 2026-03-20 16:36:41.692225 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_perses-operator-b9bc87685-dshn8_95615498-4862-4d52-9c91-6f2f4f17d4ba/perses-operator/ 2026-03-20 16:36:41.692247 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-operators_perses-operator-b9bc87685-dshn8_95615498-4862-4d52-9c91-6f2f4f17d4ba/perses-operator/0.log 2026-03-20 16:36:41.692275 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-control-plane-749d76644c-jmrs2_9c5cc732-9b60-47cf-93f1-346fa99b4ac4/ 2026-03-20 16:36:41.692298 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-control-plane-749d76644c-jmrs2_9c5cc732-9b60-47cf-93f1-346fa99b4ac4/kube-rbac-proxy/ 2026-03-20 16:36:41.692338 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-control-plane-749d76644c-jmrs2_9c5cc732-9b60-47cf-93f1-346fa99b4ac4/kube-rbac-proxy/0.log 2026-03-20 16:36:41.692363 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-control-plane-749d76644c-jmrs2_9c5cc732-9b60-47cf-93f1-346fa99b4ac4/ovnkube-cluster-manager/ 2026-03-20 16:36:41.692386 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-control-plane-749d76644c-jmrs2_9c5cc732-9b60-47cf-93f1-346fa99b4ac4/ovnkube-cluster-manager/0.log 2026-03-20 16:36:41.692409 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-9dxkv_95b45441-da8b-429c-ba9a-f4b621be3d37/ 2026-03-20 16:36:41.692433 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-9dxkv_95b45441-da8b-429c-ba9a-f4b621be3d37/kube-rbac-proxy-node/ 2026-03-20 16:36:41.692456 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-9dxkv_95b45441-da8b-429c-ba9a-f4b621be3d37/kube-rbac-proxy-node/0.log 2026-03-20 16:36:41.692479 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-9dxkv_95b45441-da8b-429c-ba9a-f4b621be3d37/kube-rbac-proxy-ovn-metrics/ 2026-03-20 16:36:41.692502 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-9dxkv_95b45441-da8b-429c-ba9a-f4b621be3d37/kube-rbac-proxy-ovn-metrics/0.log 2026-03-20 16:36:41.692524 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-9dxkv_95b45441-da8b-429c-ba9a-f4b621be3d37/kubecfg-setup/ 2026-03-20 16:36:41.692547 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-9dxkv_95b45441-da8b-429c-ba9a-f4b621be3d37/kubecfg-setup/0.log 2026-03-20 16:36:41.692570 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-9dxkv_95b45441-da8b-429c-ba9a-f4b621be3d37/nbdb/ 2026-03-20 16:36:41.692592 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-9dxkv_95b45441-da8b-429c-ba9a-f4b621be3d37/nbdb/0.log 2026-03-20 16:36:41.692615 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-9dxkv_95b45441-da8b-429c-ba9a-f4b621be3d37/northd/ 2026-03-20 16:36:41.692638 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-9dxkv_95b45441-da8b-429c-ba9a-f4b621be3d37/northd/0.log 2026-03-20 16:36:41.692660 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-9dxkv_95b45441-da8b-429c-ba9a-f4b621be3d37/ovn-acl-logging/ 2026-03-20 16:36:41.692708 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-9dxkv_95b45441-da8b-429c-ba9a-f4b621be3d37/ovn-acl-logging/0.log 2026-03-20 16:36:41.692732 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-9dxkv_95b45441-da8b-429c-ba9a-f4b621be3d37/ovn-controller/ 2026-03-20 16:36:41.692760 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-9dxkv_95b45441-da8b-429c-ba9a-f4b621be3d37/ovn-controller/0.log 2026-03-20 16:36:41.692784 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-9dxkv_95b45441-da8b-429c-ba9a-f4b621be3d37/ovnkube-controller/ 2026-03-20 16:36:41.692807 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-9dxkv_95b45441-da8b-429c-ba9a-f4b621be3d37/ovnkube-controller/0.log.gz 2026-03-20 16:36:41.692829 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-9dxkv_95b45441-da8b-429c-ba9a-f4b621be3d37/sbdb/ 2026-03-20 16:36:41.692852 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-ovn-kubernetes_ovnkube-node-9dxkv_95b45441-da8b-429c-ba9a-f4b621be3d37/sbdb/0.log 2026-03-20 16:36:41.692875 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-route-controller-manager_route-controller-manager-6f8d66c474-n6cpz_d2339af9-dd9c-4eb9-9774-2209e9b46f5e/ 2026-03-20 16:36:41.692898 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-route-controller-manager_route-controller-manager-6f8d66c474-n6cpz_d2339af9-dd9c-4eb9-9774-2209e9b46f5e/route-controller-manager/ 2026-03-20 16:36:41.692922 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-route-controller-manager_route-controller-manager-6f8d66c474-n6cpz_d2339af9-dd9c-4eb9-9774-2209e9b46f5e/route-controller-manager/0.log 2026-03-20 16:36:41.692945 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-service-ca-operator_service-ca-operator-777779d784-4b2zc_2fad111f-321a-4c9b-a945-ef93e6f95efb/ 2026-03-20 16:36:41.692967 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-service-ca-operator_service-ca-operator-777779d784-4b2zc_2fad111f-321a-4c9b-a945-ef93e6f95efb/service-ca-operator/ 2026-03-20 16:36:41.692990 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-service-ca-operator_service-ca-operator-777779d784-4b2zc_2fad111f-321a-4c9b-a945-ef93e6f95efb/service-ca-operator/0.log 2026-03-20 16:36:41.693013 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-service-ca_service-ca-9c57cc56f-6bsgz_750fbd37-7d78-4f5d-b78f-f8f2f09d703d/ 2026-03-20 16:36:41.693036 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-service-ca_service-ca-9c57cc56f-6bsgz_750fbd37-7d78-4f5d-b78f-f8f2f09d703d/service-ca-controller/ 2026-03-20 16:36:41.693059 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openshift-service-ca_service-ca-9c57cc56f-6bsgz_750fbd37-7d78-4f5d-b78f-f8f2f09d703d/service-ca-controller/0.log 2026-03-20 16:36:41.693082 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_6e5835fdb519dbfa6963d219a5ec71b0d73708b5df93d36b8aba1618batvcbb_5ad3b123-515a-46c7-be27-6bf55d5823d0/ 2026-03-20 16:36:41.693105 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_6e5835fdb519dbfa6963d219a5ec71b0d73708b5df93d36b8aba1618batvcbb_5ad3b123-515a-46c7-be27-6bf55d5823d0/extract/ 2026-03-20 16:36:41.693128 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_6e5835fdb519dbfa6963d219a5ec71b0d73708b5df93d36b8aba1618batvcbb_5ad3b123-515a-46c7-be27-6bf55d5823d0/extract/0.log 2026-03-20 16:36:41.693151 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_6e5835fdb519dbfa6963d219a5ec71b0d73708b5df93d36b8aba1618batvcbb_5ad3b123-515a-46c7-be27-6bf55d5823d0/pull/ 2026-03-20 16:36:41.693174 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_6e5835fdb519dbfa6963d219a5ec71b0d73708b5df93d36b8aba1618batvcbb_5ad3b123-515a-46c7-be27-6bf55d5823d0/pull/0.log 2026-03-20 16:36:41.693203 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_6e5835fdb519dbfa6963d219a5ec71b0d73708b5df93d36b8aba1618batvcbb_5ad3b123-515a-46c7-be27-6bf55d5823d0/util/ 2026-03-20 16:36:41.693227 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_6e5835fdb519dbfa6963d219a5ec71b0d73708b5df93d36b8aba1618batvcbb_5ad3b123-515a-46c7-be27-6bf55d5823d0/util/0.log 2026-03-20 16:36:41.693251 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_8def7df2b499ac483c55b958477ae70d30de43fc359ef64c96f9aee971khqb2_bfe1f060-6370-4060-8b43-4e6e95c1f018/ 2026-03-20 16:36:41.693275 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_8def7df2b499ac483c55b958477ae70d30de43fc359ef64c96f9aee971khqb2_bfe1f060-6370-4060-8b43-4e6e95c1f018/extract/ 2026-03-20 16:36:41.693301 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_8def7df2b499ac483c55b958477ae70d30de43fc359ef64c96f9aee971khqb2_bfe1f060-6370-4060-8b43-4e6e95c1f018/extract/0.log 2026-03-20 16:36:41.693325 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_8def7df2b499ac483c55b958477ae70d30de43fc359ef64c96f9aee971khqb2_bfe1f060-6370-4060-8b43-4e6e95c1f018/pull/ 2026-03-20 16:36:41.693350 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_8def7df2b499ac483c55b958477ae70d30de43fc359ef64c96f9aee971khqb2_bfe1f060-6370-4060-8b43-4e6e95c1f018/pull/0.log 2026-03-20 16:36:41.693374 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_8def7df2b499ac483c55b958477ae70d30de43fc359ef64c96f9aee971khqb2_bfe1f060-6370-4060-8b43-4e6e95c1f018/util/ 2026-03-20 16:36:41.693397 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_8def7df2b499ac483c55b958477ae70d30de43fc359ef64c96f9aee971khqb2_bfe1f060-6370-4060-8b43-4e6e95c1f018/util/0.log 2026-03-20 16:36:41.693420 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_barbican-operator-controller-manager-59bc569d95-2l2s5_7c7640f5-78e7-4dba-8900-26dc47eb640f/ 2026-03-20 16:36:41.693443 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_barbican-operator-controller-manager-59bc569d95-2l2s5_7c7640f5-78e7-4dba-8900-26dc47eb640f/manager/ 2026-03-20 16:36:41.693467 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_barbican-operator-controller-manager-59bc569d95-2l2s5_7c7640f5-78e7-4dba-8900-26dc47eb640f/manager/0.log 2026-03-20 16:36:41.693490 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_cinder-operator-controller-manager-8d58dc466-q88gq_f0d2c31f-5223-4222-afa8-ba918af23dca/ 2026-03-20 16:36:41.693513 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_cinder-operator-controller-manager-8d58dc466-q88gq_f0d2c31f-5223-4222-afa8-ba918af23dca/manager/ 2026-03-20 16:36:41.693536 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_cinder-operator-controller-manager-8d58dc466-q88gq_f0d2c31f-5223-4222-afa8-ba918af23dca/manager/0.log.gz 2026-03-20 16:36:41.693560 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_designate-operator-controller-manager-588d4d986b-lzq5s_6fdb989b-19f2-4bc2-97c6-2dfa1bb4cfbe/ 2026-03-20 16:36:41.693584 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_designate-operator-controller-manager-588d4d986b-lzq5s_6fdb989b-19f2-4bc2-97c6-2dfa1bb4cfbe/manager/ 2026-03-20 16:36:41.693607 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_designate-operator-controller-manager-588d4d986b-lzq5s_6fdb989b-19f2-4bc2-97c6-2dfa1bb4cfbe/manager/0.log 2026-03-20 16:36:41.693636 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_glance-operator-controller-manager-79df6bcc97-z8zt6_200b39b4-9995-48fc-a31f-1708526bd9d8/ 2026-03-20 16:36:41.693660 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_glance-operator-controller-manager-79df6bcc97-z8zt6_200b39b4-9995-48fc-a31f-1708526bd9d8/manager/ 2026-03-20 16:36:41.693711 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_glance-operator-controller-manager-79df6bcc97-z8zt6_200b39b4-9995-48fc-a31f-1708526bd9d8/manager/0.log 2026-03-20 16:36:41.693736 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_heat-operator-controller-manager-67dd5f86f5-mqm85_829ecd99-4a08-4965-ab30-fb30ab8e2ead/ 2026-03-20 16:36:41.693759 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_heat-operator-controller-manager-67dd5f86f5-mqm85_829ecd99-4a08-4965-ab30-fb30ab8e2ead/manager/ 2026-03-20 16:36:41.693782 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_heat-operator-controller-manager-67dd5f86f5-mqm85_829ecd99-4a08-4965-ab30-fb30ab8e2ead/manager/0.log 2026-03-20 16:36:41.693804 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_horizon-operator-controller-manager-8464cc45fb-v8t7w_be5e97ef-9ad5-4663-bbf1-69573e2eedeb/ 2026-03-20 16:36:41.693827 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_horizon-operator-controller-manager-8464cc45fb-v8t7w_be5e97ef-9ad5-4663-bbf1-69573e2eedeb/manager/ 2026-03-20 16:36:41.693851 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_horizon-operator-controller-manager-8464cc45fb-v8t7w_be5e97ef-9ad5-4663-bbf1-69573e2eedeb/manager/0.log 2026-03-20 16:36:41.693874 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_infra-operator-controller-manager-7b9c774f96-qzktk_bcb8373c-ae7c-4646-a90a-fe965f70c9bd/ 2026-03-20 16:36:41.693897 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_infra-operator-controller-manager-7b9c774f96-qzktk_bcb8373c-ae7c-4646-a90a-fe965f70c9bd/manager/ 2026-03-20 16:36:41.693919 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_infra-operator-controller-manager-7b9c774f96-qzktk_bcb8373c-ae7c-4646-a90a-fe965f70c9bd/manager/0.log 2026-03-20 16:36:41.693942 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_ironic-operator-controller-manager-6f787dddc9-r7v2l_7f918684-13af-4141-a414-9ac7b87e75d9/ 2026-03-20 16:36:41.693966 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_ironic-operator-controller-manager-6f787dddc9-r7v2l_7f918684-13af-4141-a414-9ac7b87e75d9/manager/ 2026-03-20 16:36:41.693989 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_ironic-operator-controller-manager-6f787dddc9-r7v2l_7f918684-13af-4141-a414-9ac7b87e75d9/manager/0.log 2026-03-20 16:36:41.694012 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_keystone-operator-controller-manager-768b96df4c-kvw9r_35420d07-3f39-47f7-bc13-d5fc95954674/ 2026-03-20 16:36:41.694035 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_keystone-operator-controller-manager-768b96df4c-kvw9r_35420d07-3f39-47f7-bc13-d5fc95954674/manager/ 2026-03-20 16:36:41.694157 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_keystone-operator-controller-manager-768b96df4c-kvw9r_35420d07-3f39-47f7-bc13-d5fc95954674/manager/0.log 2026-03-20 16:36:41.694192 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_manila-operator-controller-manager-55f864c847-6xnn6_94489fb6-8195-4820-b4bf-87122803836a/ 2026-03-20 16:36:41.694219 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_manila-operator-controller-manager-55f864c847-6xnn6_94489fb6-8195-4820-b4bf-87122803836a/manager/ 2026-03-20 16:36:41.694243 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_manila-operator-controller-manager-55f864c847-6xnn6_94489fb6-8195-4820-b4bf-87122803836a/manager/0.log 2026-03-20 16:36:41.694268 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_mariadb-operator-controller-manager-67ccfc9778-cff9z_a314b9f5-f2d9-445c-bf5d-a42dc479c21d/ 2026-03-20 16:36:41.694292 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_mariadb-operator-controller-manager-67ccfc9778-cff9z_a314b9f5-f2d9-445c-bf5d-a42dc479c21d/manager/ 2026-03-20 16:36:41.694316 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_mariadb-operator-controller-manager-67ccfc9778-cff9z_a314b9f5-f2d9-445c-bf5d-a42dc479c21d/manager/0.log 2026-03-20 16:36:41.694340 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_neutron-operator-controller-manager-767865f676-fstk4_aee070f5-c22b-4b69-b116-685316825aaa/ 2026-03-20 16:36:41.694363 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_neutron-operator-controller-manager-767865f676-fstk4_aee070f5-c22b-4b69-b116-685316825aaa/manager/ 2026-03-20 16:36:41.694387 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_neutron-operator-controller-manager-767865f676-fstk4_aee070f5-c22b-4b69-b116-685316825aaa/manager/0.log 2026-03-20 16:36:41.694410 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_nova-operator-controller-manager-5d488d59fb-vj727_6c1e7d53-7b72-4c38-bca1-94db6fd742d2/ 2026-03-20 16:36:41.694433 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_nova-operator-controller-manager-5d488d59fb-vj727_6c1e7d53-7b72-4c38-bca1-94db6fd742d2/manager/ 2026-03-20 16:36:41.694457 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_nova-operator-controller-manager-5d488d59fb-vj727_6c1e7d53-7b72-4c38-bca1-94db6fd742d2/manager/0.log 2026-03-20 16:36:41.694480 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_octavia-operator-controller-manager-5b9f45d989-rncs2_e36b96c5-7bfb-4657-b69b-7eaeeff3b477/ 2026-03-20 16:36:41.694503 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_octavia-operator-controller-manager-5b9f45d989-rncs2_e36b96c5-7bfb-4657-b69b-7eaeeff3b477/manager/ 2026-03-20 16:36:41.694526 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_octavia-operator-controller-manager-5b9f45d989-rncs2_e36b96c5-7bfb-4657-b69b-7eaeeff3b477/manager/0.log 2026-03-20 16:36:41.694549 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_openstack-baremetal-operator-controller-manager-89d64c458-ntg7z_7a422362-8c7e-4943-ad13-ca4089978ef9/ 2026-03-20 16:36:41.694572 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_openstack-baremetal-operator-controller-manager-89d64c458-ntg7z_7a422362-8c7e-4943-ad13-ca4089978ef9/manager/ 2026-03-20 16:36:41.694596 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_openstack-baremetal-operator-controller-manager-89d64c458-ntg7z_7a422362-8c7e-4943-ad13-ca4089978ef9/manager/0.log 2026-03-20 16:36:41.694625 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_openstack-operator-controller-manager-6f58c59cbb-66v95_94bef8a0-0916-46e0-9253-8130ef2367da/ 2026-03-20 16:36:41.694650 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_openstack-operator-controller-manager-6f58c59cbb-66v95_94bef8a0-0916-46e0-9253-8130ef2367da/manager/ 2026-03-20 16:36:41.694700 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_openstack-operator-controller-manager-6f58c59cbb-66v95_94bef8a0-0916-46e0-9253-8130ef2367da/manager/0.log.gz 2026-03-20 16:36:41.694727 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_openstack-operator-index-46f5x_96e2a06f-6d79-4e77-af9f-a5bc8959477a/ 2026-03-20 16:36:41.694751 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_openstack-operator-index-46f5x_96e2a06f-6d79-4e77-af9f-a5bc8959477a/registry-server/ 2026-03-20 16:36:41.694774 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_openstack-operator-index-46f5x_96e2a06f-6d79-4e77-af9f-a5bc8959477a/registry-server/0.log 2026-03-20 16:36:41.694798 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_ovn-operator-controller-manager-884679f54-xvk2s_3e11ecc5-9159-4144-a583-1b657f2349ea/ 2026-03-20 16:36:41.694821 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_ovn-operator-controller-manager-884679f54-xvk2s_3e11ecc5-9159-4144-a583-1b657f2349ea/manager/ 2026-03-20 16:36:41.694843 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_ovn-operator-controller-manager-884679f54-xvk2s_3e11ecc5-9159-4144-a583-1b657f2349ea/manager/0.log 2026-03-20 16:36:41.694866 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_placement-operator-controller-manager-5784578c99-gpxm6_1847c24e-09bc-44da-9343-5ae3f93c1dd1/ 2026-03-20 16:36:41.694889 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_placement-operator-controller-manager-5784578c99-gpxm6_1847c24e-09bc-44da-9343-5ae3f93c1dd1/manager/ 2026-03-20 16:36:41.694912 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_placement-operator-controller-manager-5784578c99-gpxm6_1847c24e-09bc-44da-9343-5ae3f93c1dd1/manager/0.log 2026-03-20 16:36:41.694934 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_rabbitmq-cluster-operator-manager-668c99d594-jmwcp_d09f9f03-5f5c-4b17-b4b7-81ddc051aef3/ 2026-03-20 16:36:41.694957 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_rabbitmq-cluster-operator-manager-668c99d594-jmwcp_d09f9f03-5f5c-4b17-b4b7-81ddc051aef3/operator/ 2026-03-20 16:36:41.694981 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_rabbitmq-cluster-operator-manager-668c99d594-jmwcp_d09f9f03-5f5c-4b17-b4b7-81ddc051aef3/operator/0.log 2026-03-20 16:36:41.695004 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_swift-operator-controller-manager-c674c5965-nqn84_a1747eb3-27f1-4f93-85c4-d786b8730bc9/ 2026-03-20 16:36:41.695028 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_swift-operator-controller-manager-c674c5965-nqn84_a1747eb3-27f1-4f93-85c4-d786b8730bc9/manager/ 2026-03-20 16:36:41.695051 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_swift-operator-controller-manager-c674c5965-nqn84_a1747eb3-27f1-4f93-85c4-d786b8730bc9/manager/0.log 2026-03-20 16:36:41.695075 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_telemetry-operator-controller-manager-d6b694c5-2xj66_31acb64e-576e-4b51-a8a3-37162d9161c0/ 2026-03-20 16:36:41.695104 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_telemetry-operator-controller-manager-d6b694c5-2xj66_31acb64e-576e-4b51-a8a3-37162d9161c0/manager/ 2026-03-20 16:36:41.695128 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_telemetry-operator-controller-manager-d6b694c5-2xj66_31acb64e-576e-4b51-a8a3-37162d9161c0/manager/0.log.gz 2026-03-20 16:36:41.695151 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_test-operator-controller-manager-5c5cb9c4d7-5rxh5_2ce265a6-3022-420a-99dc-0cead55c568f/ 2026-03-20 16:36:41.695174 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_test-operator-controller-manager-5c5cb9c4d7-5rxh5_2ce265a6-3022-420a-99dc-0cead55c568f/manager/ 2026-03-20 16:36:41.695196 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_test-operator-controller-manager-5c5cb9c4d7-5rxh5_2ce265a6-3022-420a-99dc-0cead55c568f/manager/0.log 2026-03-20 16:36:41.695219 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_watcher-operator-controller-manager-77c9f8cb5b-z626z_19a528ae-3ae2-48a5-88cb-04a92b073043/ 2026-03-20 16:36:41.695241 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_watcher-operator-controller-manager-77c9f8cb5b-z626z_19a528ae-3ae2-48a5-88cb-04a92b073043/manager/ 2026-03-20 16:36:41.695264 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_watcher-operator-controller-manager-77c9f8cb5b-z626z_19a528ae-3ae2-48a5-88cb-04a92b073043/manager/0.log.gz 2026-03-20 16:36:41.695287 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_watcher-operator-index-5c49m_a75a77fe-87c9-4704-8698-f6b526264fea/ 2026-03-20 16:36:41.695310 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_watcher-operator-index-5c49m_a75a77fe-87c9-4704-8698-f6b526264fea/registry-server/ 2026-03-20 16:36:41.695334 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/openstack-operators_watcher-operator-index-5c49m_a75a77fe-87c9-4704-8698-f6b526264fea/registry-server/0.log 2026-03-20 16:36:41.695358 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_alertmanager-metric-storage-0_6dcc5c20-eb16-44c2-be60-17a397527235/ 2026-03-20 16:36:41.695383 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_alertmanager-metric-storage-0_6dcc5c20-eb16-44c2-be60-17a397527235/alertmanager/ 2026-03-20 16:36:41.695407 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_alertmanager-metric-storage-0_6dcc5c20-eb16-44c2-be60-17a397527235/alertmanager/0.log 2026-03-20 16:36:41.695431 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_alertmanager-metric-storage-0_6dcc5c20-eb16-44c2-be60-17a397527235/config-reloader/ 2026-03-20 16:36:41.695454 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_alertmanager-metric-storage-0_6dcc5c20-eb16-44c2-be60-17a397527235/config-reloader/0.log 2026-03-20 16:36:41.695478 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_alertmanager-metric-storage-0_6dcc5c20-eb16-44c2-be60-17a397527235/init-config-reloader/ 2026-03-20 16:36:41.695501 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_alertmanager-metric-storage-0_6dcc5c20-eb16-44c2-be60-17a397527235/init-config-reloader/0.log 2026-03-20 16:36:41.695525 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_ceilometer-0_112da3cf-6ba4-41d9-b885-88df5555e526/ 2026-03-20 16:36:41.695553 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_ceilometer-0_112da3cf-6ba4-41d9-b885-88df5555e526/ceilometer-central-agent/ 2026-03-20 16:36:41.695577 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_ceilometer-0_112da3cf-6ba4-41d9-b885-88df5555e526/ceilometer-central-agent/0.log 2026-03-20 16:36:41.695600 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_ceilometer-0_112da3cf-6ba4-41d9-b885-88df5555e526/ceilometer-notification-agent/ 2026-03-20 16:36:41.695623 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_ceilometer-0_112da3cf-6ba4-41d9-b885-88df5555e526/ceilometer-notification-agent/0.log 2026-03-20 16:36:41.695646 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_ceilometer-0_112da3cf-6ba4-41d9-b885-88df5555e526/proxy-httpd/ 2026-03-20 16:36:41.695694 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_ceilometer-0_112da3cf-6ba4-41d9-b885-88df5555e526/proxy-httpd/0.log 2026-03-20 16:36:41.695720 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_ceilometer-0_112da3cf-6ba4-41d9-b885-88df5555e526/sg-core/ 2026-03-20 16:36:41.695744 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_ceilometer-0_112da3cf-6ba4-41d9-b885-88df5555e526/sg-core/0.log 2026-03-20 16:36:41.695767 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_keystone-5d69f9588b-zsrh7_488cca2e-7391-4ebb-9f9d-e6d5f04e94c4/ 2026-03-20 16:36:41.695790 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_keystone-5d69f9588b-zsrh7_488cca2e-7391-4ebb-9f9d-e6d5f04e94c4/keystone-api/ 2026-03-20 16:36:41.695813 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_keystone-5d69f9588b-zsrh7_488cca2e-7391-4ebb-9f9d-e6d5f04e94c4/keystone-api/0.log 2026-03-20 16:36:41.695837 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_keystone-cron-29567041-94xtk_d3a49c21-5df2-4013-b93e-8314741903e6/ 2026-03-20 16:36:41.695860 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_keystone-cron-29567041-94xtk_d3a49c21-5df2-4013-b93e-8314741903e6/keystone-cron/ 2026-03-20 16:36:41.695883 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_keystone-cron-29567041-94xtk_d3a49c21-5df2-4013-b93e-8314741903e6/keystone-cron/0.log 2026-03-20 16:36:41.695906 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_kube-state-metrics-0_c5449776-80e0-4686-bd0b-2fd27d88ed52/ 2026-03-20 16:36:41.695929 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_kube-state-metrics-0_c5449776-80e0-4686-bd0b-2fd27d88ed52/kube-state-metrics/ 2026-03-20 16:36:41.695952 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_kube-state-metrics-0_c5449776-80e0-4686-bd0b-2fd27d88ed52/kube-state-metrics/0.log 2026-03-20 16:36:41.695975 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_memcached-0_c4df32c5-10f9-4047-9224-c7ecb941a55f/ 2026-03-20 16:36:41.695998 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_memcached-0_c4df32c5-10f9-4047-9224-c7ecb941a55f/memcached/ 2026-03-20 16:36:41.696079 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_memcached-0_c4df32c5-10f9-4047-9224-c7ecb941a55f/memcached/0.log.gz 2026-03-20 16:36:41.696108 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_openstack-galera-0_3367f79a-2519-4c2d-8563-fac811678ed3/ 2026-03-20 16:36:41.696139 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_openstack-galera-0_3367f79a-2519-4c2d-8563-fac811678ed3/galera/ 2026-03-20 16:36:41.696163 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_openstack-galera-0_3367f79a-2519-4c2d-8563-fac811678ed3/galera/0.log 2026-03-20 16:36:41.696187 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_openstack-galera-0_3367f79a-2519-4c2d-8563-fac811678ed3/mysql-bootstrap/ 2026-03-20 16:36:41.696210 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_openstack-galera-0_3367f79a-2519-4c2d-8563-fac811678ed3/mysql-bootstrap/0.log 2026-03-20 16:36:41.696233 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_openstackclient_8cf25919-005b-4b34-98eb-28cacad075b4/ 2026-03-20 16:36:41.696256 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_openstackclient_8cf25919-005b-4b34-98eb-28cacad075b4/openstackclient/ 2026-03-20 16:36:41.696279 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_openstackclient_8cf25919-005b-4b34-98eb-28cacad075b4/openstackclient/0.log 2026-03-20 16:36:41.696301 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_prometheus-metric-storage-0_d2f088cf-8304-46b5-af82-96110c742638/ 2026-03-20 16:36:41.696324 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_prometheus-metric-storage-0_d2f088cf-8304-46b5-af82-96110c742638/config-reloader/ 2026-03-20 16:36:41.696365 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_prometheus-metric-storage-0_d2f088cf-8304-46b5-af82-96110c742638/config-reloader/0.log 2026-03-20 16:36:41.696388 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_prometheus-metric-storage-0_d2f088cf-8304-46b5-af82-96110c742638/init-config-reloader/ 2026-03-20 16:36:41.696411 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_prometheus-metric-storage-0_d2f088cf-8304-46b5-af82-96110c742638/init-config-reloader/0.log 2026-03-20 16:36:41.696434 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_prometheus-metric-storage-0_d2f088cf-8304-46b5-af82-96110c742638/prometheus/ 2026-03-20 16:36:41.696457 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_prometheus-metric-storage-0_d2f088cf-8304-46b5-af82-96110c742638/prometheus/0.log 2026-03-20 16:36:41.696480 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_prometheus-metric-storage-0_d2f088cf-8304-46b5-af82-96110c742638/thanos-sidecar/ 2026-03-20 16:36:41.696502 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_prometheus-metric-storage-0_d2f088cf-8304-46b5-af82-96110c742638/thanos-sidecar/0.log 2026-03-20 16:36:41.696525 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-notifications-server-0_0f7bff9e-8c2a-478e-a30e-55d5be1df762/ 2026-03-20 16:36:41.696548 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-notifications-server-0_0f7bff9e-8c2a-478e-a30e-55d5be1df762/rabbitmq/ 2026-03-20 16:36:41.696570 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-notifications-server-0_0f7bff9e-8c2a-478e-a30e-55d5be1df762/rabbitmq/0.log 2026-03-20 16:36:41.696593 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-notifications-server-0_0f7bff9e-8c2a-478e-a30e-55d5be1df762/setup-container/ 2026-03-20 16:36:41.696622 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-notifications-server-0_0f7bff9e-8c2a-478e-a30e-55d5be1df762/setup-container/0.log 2026-03-20 16:36:41.696645 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-server-0_ec1149a7-ab74-4cc6-9e54-66a6136d41ac/ 2026-03-20 16:36:41.696701 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-server-0_ec1149a7-ab74-4cc6-9e54-66a6136d41ac/rabbitmq/ 2026-03-20 16:36:41.696733 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-server-0_ec1149a7-ab74-4cc6-9e54-66a6136d41ac/rabbitmq/0.log 2026-03-20 16:36:41.696758 | controller | cd+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-server-0_ec1149a7-ab74-4cc6-9e54-66a6136d41ac/setup-container/ 2026-03-20 16:36:41.696782 | controller | >f+++++++++ ci-framework-data/logs/crc/crc-logs-artifacts/pods/watcher-kuttl-default_rabbitmq-server-0_ec1149a7-ab74-4cc6-9e54-66a6136d41ac/setup-container/0.log 2026-03-20 16:36:41.696807 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/ 2026-03-20 16:36:41.696831 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/event-filter.html 2026-03-20 16:36:41.696856 | controller | cL+++++++++ ci-framework-data/logs/openstack-must-gather/latest -> quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339 2026-03-20 16:36:41.696879 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/must-gather.logs 2026-03-20 16:36:41.696904 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/timestamp 2026-03-20 16:36:41.696927 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/ 2026-03-20 16:36:41.696950 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/all-containers-cpu-top.log 2026-03-20 16:36:41.696973 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/gather.logs 2026-03-20 16:36:41.696996 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/must-gather.tar.xz 2026-03-20 16:36:41.697019 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/packagemanifests.gz 2026-03-20 16:36:41.697042 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/run.log 2026-03-20 16:36:41.697066 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/version 2026-03-20 16:36:41.697089 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/apiservices/ 2026-03-20 16:36:41.697115 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/apiservices/v1beta1.barbican.openstack.org.yaml 2026-03-20 16:36:41.697147 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/apiservices/v1beta1.baremetal.openstack.org.yaml 2026-03-20 16:36:41.697172 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/apiservices/v1beta1.cinder.openstack.org.yaml 2026-03-20 16:36:41.697196 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/apiservices/v1beta1.client.openstack.org.yaml 2026-03-20 16:36:41.697220 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/apiservices/v1beta1.core.openstack.org.yaml 2026-03-20 16:36:41.697243 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/apiservices/v1beta1.dataplane.openstack.org.yaml 2026-03-20 16:36:41.697267 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/apiservices/v1beta1.designate.openstack.org.yaml 2026-03-20 16:36:41.697290 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/apiservices/v1beta1.glance.openstack.org.yaml 2026-03-20 16:36:41.697314 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/apiservices/v1beta1.heat.openstack.org.yaml 2026-03-20 16:36:41.697337 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/apiservices/v1beta1.horizon.openstack.org.yaml 2026-03-20 16:36:41.697360 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/apiservices/v1beta1.instanceha.openstack.org.yaml 2026-03-20 16:36:41.697384 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/apiservices/v1beta1.ironic.openstack.org.yaml 2026-03-20 16:36:41.697407 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/apiservices/v1beta1.keystone.openstack.org.yaml 2026-03-20 16:36:41.697431 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/apiservices/v1beta1.manila.openstack.org.yaml 2026-03-20 16:36:41.697454 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/apiservices/v1beta1.mariadb.openstack.org.yaml 2026-03-20 16:36:41.697478 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/apiservices/v1beta1.memcached.openstack.org.yaml 2026-03-20 16:36:41.697507 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/apiservices/v1beta1.network.openstack.org.yaml 2026-03-20 16:36:41.697532 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/apiservices/v1beta1.neutron.openstack.org.yaml 2026-03-20 16:36:41.697556 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/apiservices/v1beta1.nova.openstack.org.yaml 2026-03-20 16:36:41.697579 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/apiservices/v1beta1.octavia.openstack.org.yaml 2026-03-20 16:36:41.697603 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/apiservices/v1beta1.operator.openstack.org.yaml 2026-03-20 16:36:41.697626 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/apiservices/v1beta1.ovn.openstack.org.yaml 2026-03-20 16:36:41.697649 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/apiservices/v1beta1.placement.openstack.org.yaml 2026-03-20 16:36:41.697700 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/apiservices/v1beta1.rabbitmq.com.yaml 2026-03-20 16:36:41.697726 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/apiservices/v1beta1.rabbitmq.openstack.org.yaml 2026-03-20 16:36:41.697750 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/apiservices/v1beta1.redis.openstack.org.yaml 2026-03-20 16:36:41.697774 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/apiservices/v1beta1.swift.openstack.org.yaml 2026-03-20 16:36:41.697798 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/apiservices/v1beta1.telemetry.openstack.org.yaml 2026-03-20 16:36:41.697821 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/apiservices/v1beta1.test.openstack.org.yaml 2026-03-20 16:36:41.697844 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/apiservices/v1beta1.topology.openstack.org.yaml 2026-03-20 16:36:41.697868 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/apiservices/v1beta1.watcher.openstack.org.yaml 2026-03-20 16:36:41.697897 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/ 2026-03-20 16:36:41.697921 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/ansibletests.test.openstack.org.yaml 2026-03-20 16:36:41.697944 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/autoscalings.telemetry.openstack.org.yaml 2026-03-20 16:36:41.697968 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/barbicanapis.barbican.openstack.org.yaml 2026-03-20 16:36:41.697991 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/barbicankeystonelisteners.barbican.openstack.org.yaml 2026-03-20 16:36:41.698014 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/barbicans.barbican.openstack.org.yaml 2026-03-20 16:36:41.698037 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/barbicanworkers.barbican.openstack.org.yaml 2026-03-20 16:36:41.698061 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/bgpconfigurations.network.openstack.org.yaml 2026-03-20 16:36:41.698084 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/ceilometers.telemetry.openstack.org.yaml 2026-03-20 16:36:41.698108 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/cinderapis.cinder.openstack.org.yaml 2026-03-20 16:36:41.698131 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/cinderbackups.cinder.openstack.org.yaml 2026-03-20 16:36:41.698154 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/cinders.cinder.openstack.org.yaml 2026-03-20 16:36:41.698177 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/cinderschedulers.cinder.openstack.org.yaml 2026-03-20 16:36:41.698199 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/cindervolumes.cinder.openstack.org.yaml 2026-03-20 16:36:41.698222 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/cloudkitties.telemetry.openstack.org.yaml 2026-03-20 16:36:41.698244 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/cloudkittyapis.telemetry.openstack.org.yaml 2026-03-20 16:36:41.698278 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/cloudkittyprocs.telemetry.openstack.org.yaml 2026-03-20 16:36:41.698302 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/designateapis.designate.openstack.org.yaml 2026-03-20 16:36:41.698326 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/designatebackendbind9s.designate.openstack.org.yaml 2026-03-20 16:36:41.698349 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/designatecentrals.designate.openstack.org.yaml 2026-03-20 16:36:41.698373 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/designatemdnses.designate.openstack.org.yaml 2026-03-20 16:36:41.698397 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/designateproducers.designate.openstack.org.yaml 2026-03-20 16:36:41.698421 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/designates.designate.openstack.org.yaml 2026-03-20 16:36:41.698445 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/designateunbounds.designate.openstack.org.yaml 2026-03-20 16:36:41.698469 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/designateworkers.designate.openstack.org.yaml 2026-03-20 16:36:41.698492 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/dnsdata.network.openstack.org.yaml 2026-03-20 16:36:41.698516 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/dnsmasqs.network.openstack.org.yaml 2026-03-20 16:36:41.698539 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/galerabackups.mariadb.openstack.org.yaml 2026-03-20 16:36:41.698562 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/galerarestores.mariadb.openstack.org.yaml 2026-03-20 16:36:41.698585 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/galeras.mariadb.openstack.org.yaml 2026-03-20 16:36:41.698607 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/glanceapis.glance.openstack.org.yaml 2026-03-20 16:36:41.698636 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/glances.glance.openstack.org.yaml 2026-03-20 16:36:41.698659 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/heatapis.heat.openstack.org.yaml 2026-03-20 16:36:41.698709 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/heatcfnapis.heat.openstack.org.yaml 2026-03-20 16:36:41.698733 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/heatengines.heat.openstack.org.yaml 2026-03-20 16:36:41.698757 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/heats.heat.openstack.org.yaml 2026-03-20 16:36:41.698780 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/horizons.horizon.openstack.org.yaml 2026-03-20 16:36:41.698803 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/horizontests.test.openstack.org.yaml 2026-03-20 16:36:41.698827 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/instancehas.instanceha.openstack.org.yaml 2026-03-20 16:36:41.698850 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/ipsets.network.openstack.org.yaml 2026-03-20 16:36:41.698874 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/ironicapis.ironic.openstack.org.yaml 2026-03-20 16:36:41.698898 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/ironicconductors.ironic.openstack.org.yaml 2026-03-20 16:36:41.698921 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/ironicinspectors.ironic.openstack.org.yaml 2026-03-20 16:36:41.698945 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/ironicneutronagents.ironic.openstack.org.yaml 2026-03-20 16:36:41.698967 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/ironics.ironic.openstack.org.yaml 2026-03-20 16:36:41.698990 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/keystoneapis.keystone.openstack.org.yaml 2026-03-20 16:36:41.699019 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/keystoneapplicationcredentials.keystone.openstack.org.yaml 2026-03-20 16:36:41.699043 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/keystoneendpoints.keystone.openstack.org.yaml 2026-03-20 16:36:41.699066 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/keystoneservices.keystone.openstack.org.yaml 2026-03-20 16:36:41.699089 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/loggings.telemetry.openstack.org.yaml 2026-03-20 16:36:41.699112 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/manilaapis.manila.openstack.org.yaml 2026-03-20 16:36:41.699135 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/manilas.manila.openstack.org.yaml 2026-03-20 16:36:41.699158 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/manilaschedulers.manila.openstack.org.yaml 2026-03-20 16:36:41.699181 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/manilashares.manila.openstack.org.yaml 2026-03-20 16:36:41.699204 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/mariadbaccounts.mariadb.openstack.org.yaml 2026-03-20 16:36:41.699226 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/mariadbdatabases.mariadb.openstack.org.yaml 2026-03-20 16:36:41.699249 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/memcacheds.memcached.openstack.org.yaml 2026-03-20 16:36:41.699272 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/metricstorages.telemetry.openstack.org.yaml 2026-03-20 16:36:41.699294 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/netconfigs.network.openstack.org.yaml 2026-03-20 16:36:41.699317 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/neutronapis.neutron.openstack.org.yaml 2026-03-20 16:36:41.699340 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/nova.nova.openstack.org.yaml 2026-03-20 16:36:41.699368 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/novaapis.nova.openstack.org.yaml 2026-03-20 16:36:41.699391 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/novacells.nova.openstack.org.yaml 2026-03-20 16:36:41.699414 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/novacomputes.nova.openstack.org.yaml 2026-03-20 16:36:41.699436 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/novaconductors.nova.openstack.org.yaml 2026-03-20 16:36:41.699459 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/novametadata.nova.openstack.org.yaml 2026-03-20 16:36:41.699482 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/novanovncproxies.nova.openstack.org.yaml 2026-03-20 16:36:41.699504 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/novaschedulers.nova.openstack.org.yaml 2026-03-20 16:36:41.699527 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/octaviaamphoracontrollers.octavia.openstack.org.yaml 2026-03-20 16:36:41.699550 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/octaviaapis.octavia.openstack.org.yaml 2026-03-20 16:36:41.699574 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/octaviarsyslogs.octavia.openstack.org.yaml 2026-03-20 16:36:41.699598 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/octavias.octavia.openstack.org.yaml 2026-03-20 16:36:41.699621 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/openstackbaremetalsets.baremetal.openstack.org.yaml 2026-03-20 16:36:41.699644 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/openstackclients.client.openstack.org.yaml 2026-03-20 16:36:41.699688 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/openstackcontrolplanes.core.openstack.org.yaml 2026-03-20 16:36:41.699718 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/openstackdataplanedeployments.dataplane.openstack.org.yaml 2026-03-20 16:36:41.699748 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/openstackdataplanenodesets.dataplane.openstack.org.yaml 2026-03-20 16:36:41.699772 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/openstackdataplaneservices.dataplane.openstack.org.yaml 2026-03-20 16:36:41.699796 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/openstackprovisionservers.baremetal.openstack.org.yaml 2026-03-20 16:36:41.699819 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/openstacks.operator.openstack.org.yaml 2026-03-20 16:36:41.699842 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/openstackversions.core.openstack.org.yaml 2026-03-20 16:36:41.699866 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/ovncontrollers.ovn.openstack.org.yaml 2026-03-20 16:36:41.699889 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/ovndbclusters.ovn.openstack.org.yaml 2026-03-20 16:36:41.699912 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/ovnnorthds.ovn.openstack.org.yaml 2026-03-20 16:36:41.699935 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/placementapis.placement.openstack.org.yaml 2026-03-20 16:36:41.699958 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/rabbitmqclusters.rabbitmq.com.yaml 2026-03-20 16:36:41.699980 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/rabbitmqpolicies.rabbitmq.openstack.org.yaml 2026-03-20 16:36:41.700004 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/rabbitmqs.rabbitmq.openstack.org.yaml 2026-03-20 16:36:41.700027 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/rabbitmqusers.rabbitmq.openstack.org.yaml 2026-03-20 16:36:41.700050 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/rabbitmqvhosts.rabbitmq.openstack.org.yaml 2026-03-20 16:36:41.700073 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/redises.redis.openstack.org.yaml 2026-03-20 16:36:41.700096 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/reservations.network.openstack.org.yaml 2026-03-20 16:36:41.700134 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/swiftproxies.swift.openstack.org.yaml 2026-03-20 16:36:41.700159 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/swiftrings.swift.openstack.org.yaml 2026-03-20 16:36:41.700182 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/swifts.swift.openstack.org.yaml 2026-03-20 16:36:41.700206 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/swiftstorages.swift.openstack.org.yaml 2026-03-20 16:36:41.700229 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/telemetries.telemetry.openstack.org.yaml 2026-03-20 16:36:41.700252 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/tempests.test.openstack.org.yaml 2026-03-20 16:36:41.700276 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/tobikoes.test.openstack.org.yaml 2026-03-20 16:36:41.700299 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/topologies.topology.openstack.org.yaml 2026-03-20 16:36:41.700322 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/transporturls.rabbitmq.openstack.org.yaml 2026-03-20 16:36:41.700362 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/watcherapis.watcher.openstack.org.yaml 2026-03-20 16:36:41.700387 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/watcherappliers.watcher.openstack.org.yaml 2026-03-20 16:36:41.700411 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/watcherdecisionengines.watcher.openstack.org.yaml 2026-03-20 16:36:41.700434 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/crd/watchers.watcher.openstack.org.yaml 2026-03-20 16:36:41.700457 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/csv/ 2026-03-20 16:36:41.700481 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/csv/cluster-observability-operator.v1.4.0 2026-03-20 16:36:41.700511 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/csv/kubernetes-nmstate-operator.4.18.0-202603041813 2026-03-20 16:36:41.700535 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/csv/metallb-operator.v4.18.0-202603040208 2026-03-20 16:36:41.700559 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/csv/openstack-operator.v0.6.0 2026-03-20 16:36:41.700583 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/csv/watcher-operator.v0.0.1 2026-03-20 16:36:41.700606 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/ctlplane/ 2026-03-20 16:36:41.700630 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/ctlplane/endpoints 2026-03-20 16:36:41.700653 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/ctlplane/services 2026-03-20 16:36:41.700699 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/ctlplane/ovn/ 2026-03-20 16:36:41.700726 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/ctlplane/ovn/cluster-status/ 2026-03-20 16:36:41.700751 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/ctlplane/ovn/cluster-status/nb/ 2026-03-20 16:36:41.700775 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/ctlplane/ovn/cluster-status/sb/ 2026-03-20 16:36:41.700799 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/ctlplane/rabbitmq/ 2026-03-20 16:36:41.700823 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/dbs/ 2026-03-20 16:36:41.700848 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/dbs/watcher-kuttl-default/ 2026-03-20 16:36:41.700874 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/ 2026-03-20 16:36:41.700898 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/ 2026-03-20 16:36:41.700928 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/all_resources.log 2026-03-20 16:36:41.700953 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/events.log 2026-03-20 16:36:41.700977 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/nad.log 2026-03-20 16:36:41.701001 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/pvc.log 2026-03-20 16:36:41.701025 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/buildconfig/ 2026-03-20 16:36:41.701049 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/cert/ 2026-03-20 16:36:41.701074 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/certificaterequests/ 2026-03-20 16:36:41.701098 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/configmaps/ 2026-03-20 16:36:41.701123 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/cronjobs/ 2026-03-20 16:36:41.701147 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/daemonset/ 2026-03-20 16:36:41.701171 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/deployments/ 2026-03-20 16:36:41.701194 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/deployments/cert-manager-cainjector.yaml 2026-03-20 16:36:41.701218 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/deployments/cert-manager-webhook.yaml 2026-03-20 16:36:41.701242 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/deployments/cert-manager.yaml 2026-03-20 16:36:41.701267 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/imagestream/ 2026-03-20 16:36:41.701292 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/installplans/ 2026-03-20 16:36:41.701322 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/issuer/ 2026-03-20 16:36:41.701348 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/jobs/ 2026-03-20 16:36:41.701373 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/poddisruptionbudgets/ 2026-03-20 16:36:41.701398 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/pods/ 2026-03-20 16:36:41.701422 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/pods/cert-manager-545d4d4674-7rt5m/ 2026-03-20 16:36:41.701447 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/pods/cert-manager-545d4d4674-7rt5m/cert-manager-545d4d4674-7rt5m-describe 2026-03-20 16:36:41.701472 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/pods/cert-manager-545d4d4674-7rt5m/logs/ 2026-03-20 16:36:41.701497 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/pods/cert-manager-545d4d4674-7rt5m/logs/cert-manager-controller.log 2026-03-20 16:36:41.701521 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/pods/cert-manager-cainjector-5545bd876-st7jn/ 2026-03-20 16:36:41.701545 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/pods/cert-manager-cainjector-5545bd876-st7jn/cert-manager-cainjector-5545bd876-st7jn-describe 2026-03-20 16:36:41.701569 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/pods/cert-manager-cainjector-5545bd876-st7jn/logs/ 2026-03-20 16:36:41.701593 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/pods/cert-manager-cainjector-5545bd876-st7jn/logs/cert-manager-cainjector.log 2026-03-20 16:36:41.701616 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/pods/cert-manager-webhook-6888856db4-n56d9/ 2026-03-20 16:36:41.701640 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/pods/cert-manager-webhook-6888856db4-n56d9/cert-manager-webhook-6888856db4-n56d9-describe 2026-03-20 16:36:41.701692 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/pods/cert-manager-webhook-6888856db4-n56d9/logs/ 2026-03-20 16:36:41.701724 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/pods/cert-manager-webhook-6888856db4-n56d9/logs/cert-manager-webhook.log 2026-03-20 16:36:41.701749 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/replicaset/ 2026-03-20 16:36:41.701774 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/replicaset/cert-manager-545d4d4674.yaml 2026-03-20 16:36:41.701798 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/replicaset/cert-manager-cainjector-5545bd876.yaml 2026-03-20 16:36:41.701823 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/replicaset/cert-manager-webhook-6888856db4.yaml 2026-03-20 16:36:41.701847 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/routes/ 2026-03-20 16:36:41.701871 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/services/ 2026-03-20 16:36:41.701895 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/services/cert-manager-cainjector.yaml 2026-03-20 16:36:41.701919 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/services/cert-manager-webhook.yaml 2026-03-20 16:36:41.701943 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/services/cert-manager.yaml 2026-03-20 16:36:41.701967 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/statefulset/ 2026-03-20 16:36:41.701991 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/cert-manager/subscriptions/ 2026-03-20 16:36:41.702014 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/ 2026-03-20 16:36:41.702043 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/all_resources.log 2026-03-20 16:36:41.702067 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/events.log 2026-03-20 16:36:41.702092 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/nad.log 2026-03-20 16:36:41.702115 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/pvc.log 2026-03-20 16:36:41.702139 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/buildconfig/ 2026-03-20 16:36:41.702162 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/cert/ 2026-03-20 16:36:41.702186 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/certificaterequests/ 2026-03-20 16:36:41.702210 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/configmaps/ 2026-03-20 16:36:41.702234 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/cronjobs/ 2026-03-20 16:36:41.702258 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/crs/ 2026-03-20 16:36:41.702282 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/crs/ipaddresspools.metallb.io/ 2026-03-20 16:36:41.702305 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/crs/ipaddresspools.metallb.io/ctlplane.yaml 2026-03-20 16:36:41.702329 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/crs/ipaddresspools.metallb.io/designateext.yaml 2026-03-20 16:36:41.702353 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/crs/ipaddresspools.metallb.io/internalapi.yaml 2026-03-20 16:36:41.702376 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/crs/ipaddresspools.metallb.io/storage.yaml 2026-03-20 16:36:41.702405 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/crs/ipaddresspools.metallb.io/tenant.yaml 2026-03-20 16:36:41.702429 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/crs/l2advertisements.metallb.io/ 2026-03-20 16:36:41.702452 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/crs/l2advertisements.metallb.io/ctlplane.yaml 2026-03-20 16:36:41.702476 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/crs/l2advertisements.metallb.io/designateext.yaml 2026-03-20 16:36:41.702499 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/crs/l2advertisements.metallb.io/internalapi.yaml 2026-03-20 16:36:41.702522 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/crs/l2advertisements.metallb.io/storage.yaml 2026-03-20 16:36:41.702545 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/crs/l2advertisements.metallb.io/tenant.yaml 2026-03-20 16:36:41.702568 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/crs/metallbs.metallb.io/ 2026-03-20 16:36:41.702591 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/crs/metallbs.metallb.io/metallb.yaml 2026-03-20 16:36:41.702614 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/daemonset/ 2026-03-20 16:36:41.702637 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/daemonset/frr-k8s.yaml 2026-03-20 16:36:41.702660 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/daemonset/speaker.yaml 2026-03-20 16:36:41.702718 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/deployments/ 2026-03-20 16:36:41.702742 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/deployments/controller.yaml 2026-03-20 16:36:41.702764 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/deployments/frr-k8s-webhook-server.yaml 2026-03-20 16:36:41.702793 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/deployments/metallb-operator-controller-manager.yaml 2026-03-20 16:36:41.702817 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/deployments/metallb-operator-webhook-server.yaml 2026-03-20 16:36:41.702840 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/imagestream/ 2026-03-20 16:36:41.702863 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/installplans/ 2026-03-20 16:36:41.702888 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/installplans/install-797pv 2026-03-20 16:36:41.702913 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/issuer/ 2026-03-20 16:36:41.702936 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/jobs/ 2026-03-20 16:36:41.702959 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/poddisruptionbudgets/ 2026-03-20 16:36:41.702982 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/pods/ 2026-03-20 16:36:41.703005 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/pods/controller-7bb4cc7c98-f45rt/ 2026-03-20 16:36:41.703028 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/pods/controller-7bb4cc7c98-f45rt/controller-7bb4cc7c98-f45rt-describe 2026-03-20 16:36:41.703051 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/pods/controller-7bb4cc7c98-f45rt/logs/ 2026-03-20 16:36:41.703075 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/pods/controller-7bb4cc7c98-f45rt/logs/controller.log.gz 2026-03-20 16:36:41.703098 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/pods/controller-7bb4cc7c98-f45rt/logs/kube-rbac-proxy.log 2026-03-20 16:36:41.703128 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/pods/frr-k8s-fbdp4/ 2026-03-20 16:36:41.703152 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/pods/frr-k8s-fbdp4/frr-k8s-fbdp4-describe 2026-03-20 16:36:41.703175 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/pods/frr-k8s-fbdp4/logs/ 2026-03-20 16:36:41.703199 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/pods/frr-k8s-fbdp4/logs/controller.log 2026-03-20 16:36:41.703222 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/pods/frr-k8s-fbdp4/logs/cp-frr-files.log 2026-03-20 16:36:41.703245 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/pods/frr-k8s-fbdp4/logs/cp-metrics.log 2026-03-20 16:36:41.703268 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/pods/frr-k8s-fbdp4/logs/cp-reloader.log 2026-03-20 16:36:41.703291 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/pods/frr-k8s-fbdp4/logs/frr-metrics.log 2026-03-20 16:36:41.703314 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/pods/frr-k8s-fbdp4/logs/frr.log.gz 2026-03-20 16:36:41.703337 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/pods/frr-k8s-fbdp4/logs/kube-rbac-proxy-frr.log 2026-03-20 16:36:41.703360 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/pods/frr-k8s-fbdp4/logs/kube-rbac-proxy.log 2026-03-20 16:36:41.703383 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/pods/frr-k8s-fbdp4/logs/reloader.log 2026-03-20 16:36:41.703406 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/pods/frr-k8s-webhook-server-bcc4b6f68-fxssk/ 2026-03-20 16:36:41.703429 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/pods/frr-k8s-webhook-server-bcc4b6f68-fxssk/frr-k8s-webhook-server-bcc4b6f68-fxssk-describe 2026-03-20 16:36:41.703459 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/pods/frr-k8s-webhook-server-bcc4b6f68-fxssk/logs/ 2026-03-20 16:36:41.703483 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/pods/frr-k8s-webhook-server-bcc4b6f68-fxssk/logs/frr-k8s-webhook-server.log 2026-03-20 16:36:41.703507 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/pods/metallb-operator-controller-manager-bfc448998-db8w6/ 2026-03-20 16:36:41.703532 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/pods/metallb-operator-controller-manager-bfc448998-db8w6/metallb-operator-controller-manager-bfc448998-db8w6-describe 2026-03-20 16:36:41.703557 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/pods/metallb-operator-controller-manager-bfc448998-db8w6/logs/ 2026-03-20 16:36:41.703581 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/pods/metallb-operator-controller-manager-bfc448998-db8w6/logs/manager.log 2026-03-20 16:36:41.703604 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/pods/metallb-operator-webhook-server-795784bfcc-vcd2s/ 2026-03-20 16:36:41.703628 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/pods/metallb-operator-webhook-server-795784bfcc-vcd2s/metallb-operator-webhook-server-795784bfcc-vcd2s-describe 2026-03-20 16:36:41.703651 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/pods/metallb-operator-webhook-server-795784bfcc-vcd2s/logs/ 2026-03-20 16:36:41.703701 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/pods/metallb-operator-webhook-server-795784bfcc-vcd2s/logs/webhook-server.log 2026-03-20 16:36:41.703727 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/pods/speaker-skmm6/ 2026-03-20 16:36:41.703751 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/pods/speaker-skmm6/speaker-skmm6-describe 2026-03-20 16:36:41.703774 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/pods/speaker-skmm6/logs/ 2026-03-20 16:36:41.703808 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/pods/speaker-skmm6/logs/kube-rbac-proxy.log 2026-03-20 16:36:41.703832 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/pods/speaker-skmm6/logs/speaker.log.gz 2026-03-20 16:36:41.703856 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/replicaset/ 2026-03-20 16:36:41.703880 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/replicaset/controller-7bb4cc7c98.yaml 2026-03-20 16:36:41.703904 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/replicaset/frr-k8s-webhook-server-bcc4b6f68.yaml 2026-03-20 16:36:41.703928 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/replicaset/metallb-operator-controller-manager-bfc448998.yaml 2026-03-20 16:36:41.703953 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/replicaset/metallb-operator-webhook-server-795784bfcc.yaml 2026-03-20 16:36:41.703977 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/routes/ 2026-03-20 16:36:41.704001 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/services/ 2026-03-20 16:36:41.704025 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/services/controller-monitor-service.yaml 2026-03-20 16:36:41.704049 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/services/frr-k8s-monitor-service.yaml 2026-03-20 16:36:41.704072 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/services/frr-k8s-webhook-service.yaml 2026-03-20 16:36:41.704096 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/services/metallb-operator-controller-manager-service.yaml 2026-03-20 16:36:41.704120 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/services/metallb-operator-webhook-server-service.yaml 2026-03-20 16:36:41.704149 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/services/metallb-operator-webhook-service.yaml 2026-03-20 16:36:41.704262 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/services/speaker-monitor-service.yaml 2026-03-20 16:36:41.704299 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/services/webhook-service.yaml 2026-03-20 16:36:41.704325 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/statefulset/ 2026-03-20 16:36:41.704350 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/subscriptions/ 2026-03-20 16:36:41.704388 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/metallb-system/subscriptions/metallb-operator-sub 2026-03-20 16:36:41.704413 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/ 2026-03-20 16:36:41.704438 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/all_resources.log 2026-03-20 16:36:41.704461 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/events.log 2026-03-20 16:36:41.704485 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/nad.log 2026-03-20 16:36:41.704509 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/pvc.log 2026-03-20 16:36:41.704532 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/buildconfig/ 2026-03-20 16:36:41.704556 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/cert/ 2026-03-20 16:36:41.704579 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/certificaterequests/ 2026-03-20 16:36:41.704602 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/configmaps/ 2026-03-20 16:36:41.704631 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/cronjobs/ 2026-03-20 16:36:41.704655 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/daemonset/ 2026-03-20 16:36:41.704709 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/deployments/ 2026-03-20 16:36:41.704734 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/deployments/control-plane-machine-set-operator.yaml 2026-03-20 16:36:41.704758 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/deployments/machine-api-operator.yaml 2026-03-20 16:36:41.704782 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/imagestream/ 2026-03-20 16:36:41.704806 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/installplans/ 2026-03-20 16:36:41.704829 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/issuer/ 2026-03-20 16:36:41.704852 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/jobs/ 2026-03-20 16:36:41.704875 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/poddisruptionbudgets/ 2026-03-20 16:36:41.704898 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/pods/ 2026-03-20 16:36:41.704918 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-78cbb6b69f-lzrhk/ 2026-03-20 16:36:41.704934 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-78cbb6b69f-lzrhk/control-plane-machine-set-operator-78cbb6b69f-lzrhk-describe 2026-03-20 16:36:41.704950 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-78cbb6b69f-lzrhk/logs/ 2026-03-20 16:36:41.704971 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/pods/control-plane-machine-set-operator-78cbb6b69f-lzrhk/logs/control-plane-machine-set-operator.log 2026-03-20 16:36:41.704988 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/pods/machine-api-operator-5694c8668f-hszjg/ 2026-03-20 16:36:41.705004 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/pods/machine-api-operator-5694c8668f-hszjg/machine-api-operator-5694c8668f-hszjg-describe 2026-03-20 16:36:41.705020 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/pods/machine-api-operator-5694c8668f-hszjg/logs/ 2026-03-20 16:36:41.705037 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/pods/machine-api-operator-5694c8668f-hszjg/logs/kube-rbac-proxy.log 2026-03-20 16:36:41.705053 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/pods/machine-api-operator-5694c8668f-hszjg/logs/machine-api-operator.log 2026-03-20 16:36:41.705069 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/replicaset/ 2026-03-20 16:36:41.705085 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/replicaset/control-plane-machine-set-operator-78cbb6b69f.yaml 2026-03-20 16:36:41.705101 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/replicaset/machine-api-operator-5694c8668f.yaml 2026-03-20 16:36:41.705117 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/routes/ 2026-03-20 16:36:41.705133 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/services/ 2026-03-20 16:36:41.705149 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/services/cluster-autoscaler-operator.yaml 2026-03-20 16:36:41.705165 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/services/control-plane-machine-set-operator.yaml 2026-03-20 16:36:41.705181 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/services/machine-api-controllers.yaml 2026-03-20 16:36:41.705202 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/services/machine-api-operator-machine-webhook.yaml 2026-03-20 16:36:41.705219 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/services/machine-api-operator-webhook.yaml 2026-03-20 16:36:41.705235 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/services/machine-api-operator.yaml 2026-03-20 16:36:41.705252 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/statefulset/ 2026-03-20 16:36:41.705268 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-machine-api/subscriptions/ 2026-03-20 16:36:41.705285 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/ 2026-03-20 16:36:41.705301 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/all_resources.log 2026-03-20 16:36:41.705317 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/events.log 2026-03-20 16:36:41.705332 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/nad.log 2026-03-20 16:36:41.705348 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pvc.log 2026-03-20 16:36:41.705363 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/buildconfig/ 2026-03-20 16:36:41.705378 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/cert/ 2026-03-20 16:36:41.705393 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/certificaterequests/ 2026-03-20 16:36:41.705408 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/configmaps/ 2026-03-20 16:36:41.705428 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/cronjobs/ 2026-03-20 16:36:41.705444 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/daemonset/ 2026-03-20 16:36:41.705459 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/deployments/ 2026-03-20 16:36:41.705474 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/deployments/marketplace-operator.yaml 2026-03-20 16:36:41.705490 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/imagestream/ 2026-03-20 16:36:41.705507 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/installplans/ 2026-03-20 16:36:41.705523 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/issuer/ 2026-03-20 16:36:41.705538 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/jobs/ 2026-03-20 16:36:41.705554 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/jobs/1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde874832f3.yaml 2026-03-20 16:36:41.705569 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/jobs/2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c166a6a.yaml 2026-03-20 16:36:41.705585 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/jobs/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e56eb0c.yaml 2026-03-20 16:36:41.705601 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/jobs/93d662022be5376a0ed3676a120a68427f47e4653a19a985adf9239726f148f.yaml 2026-03-20 16:36:41.705616 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/poddisruptionbudgets/ 2026-03-20 16:36:41.705632 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/ 2026-03-20 16:36:41.705648 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde874crv8c/ 2026-03-20 16:36:41.705684 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde874crv8c/1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde874crv8c-describe 2026-03-20 16:36:41.705710 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde874crv8c/logs/ 2026-03-20 16:36:41.705728 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde874crv8c/logs/extract.log 2026-03-20 16:36:41.705743 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde874crv8c/logs/pull.log 2026-03-20 16:36:41.705759 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/1d8741a795bd73341bdd61a6e59c08511cf9466dbb5fc4045ac2dde874crv8c/logs/util.log 2026-03-20 16:36:41.705775 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c167lkk/ 2026-03-20 16:36:41.705790 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c167lkk/2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c167lkk-describe 2026-03-20 16:36:41.705806 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c167lkk/logs/ 2026-03-20 16:36:41.705822 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c167lkk/logs/extract.log 2026-03-20 16:36:41.705837 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c167lkk/logs/pull.log 2026-03-20 16:36:41.705853 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/2d3ddce10053cc6867b5a0ce1614b30225f3a63fab79a72148165675c167lkk/logs/util.log 2026-03-20 16:36:41.705874 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5vx5n9/ 2026-03-20 16:36:41.705890 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5vx5n9/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5vx5n9-describe 2026-03-20 16:36:41.705906 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5vx5n9/logs/ 2026-03-20 16:36:41.705922 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5vx5n9/logs/extract.log 2026-03-20 16:36:41.705938 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5vx5n9/logs/pull.log 2026-03-20 16:36:41.705953 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/925ad1f05bf386dc21bdfe2f8249c1fbfd04a404dec7a7fb6362d758e5vx5n9/logs/util.log 2026-03-20 16:36:41.705969 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/93d662022be5376a0ed3676a120a68427f47e4653a19a985adf923972657c7m/ 2026-03-20 16:36:41.705984 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/93d662022be5376a0ed3676a120a68427f47e4653a19a985adf923972657c7m/93d662022be5376a0ed3676a120a68427f47e4653a19a985adf923972657c7m-describe 2026-03-20 16:36:41.706000 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/93d662022be5376a0ed3676a120a68427f47e4653a19a985adf923972657c7m/logs/ 2026-03-20 16:36:41.706016 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/93d662022be5376a0ed3676a120a68427f47e4653a19a985adf923972657c7m/logs/extract.log 2026-03-20 16:36:41.706031 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/93d662022be5376a0ed3676a120a68427f47e4653a19a985adf923972657c7m/logs/pull.log 2026-03-20 16:36:41.706047 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/93d662022be5376a0ed3676a120a68427f47e4653a19a985adf923972657c7m/logs/util.log 2026-03-20 16:36:41.706066 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/certified-operators-bfbr2/ 2026-03-20 16:36:41.706082 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/certified-operators-bfbr2/certified-operators-bfbr2-describe 2026-03-20 16:36:41.706097 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/certified-operators-bfbr2/logs/ 2026-03-20 16:36:41.706113 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/certified-operators-bfbr2/logs/extract-content.log 2026-03-20 16:36:41.706179 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/certified-operators-bfbr2/logs/extract-utilities.log 2026-03-20 16:36:41.706200 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/certified-operators-bfbr2/logs/registry-server.log.gz 2026-03-20 16:36:41.706217 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/community-operators-bwvk2/ 2026-03-20 16:36:41.706233 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/community-operators-bwvk2/community-operators-bwvk2-describe 2026-03-20 16:36:41.706249 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/community-operators-bwvk2/logs/ 2026-03-20 16:36:41.706265 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/community-operators-bwvk2/logs/extract-content.log 2026-03-20 16:36:41.706281 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/community-operators-bwvk2/logs/extract-utilities.log 2026-03-20 16:36:41.706297 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/community-operators-bwvk2/logs/registry-server.log.gz 2026-03-20 16:36:41.706312 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/marketplace-operator-79b997595-qrqgm/ 2026-03-20 16:36:41.706332 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/marketplace-operator-79b997595-qrqgm/marketplace-operator-79b997595-qrqgm-describe 2026-03-20 16:36:41.706348 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/marketplace-operator-79b997595-qrqgm/logs/ 2026-03-20 16:36:41.706364 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/marketplace-operator-79b997595-qrqgm/logs/marketplace-operator.log 2026-03-20 16:36:41.706380 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/redhat-marketplace-22jh6/ 2026-03-20 16:36:41.706397 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/redhat-marketplace-22jh6/redhat-marketplace-22jh6-describe 2026-03-20 16:36:41.706413 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/redhat-marketplace-22jh6/logs/ 2026-03-20 16:36:41.706429 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/redhat-marketplace-22jh6/logs/extract-content.log 2026-03-20 16:36:41.706445 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/redhat-marketplace-22jh6/logs/extract-utilities.log 2026-03-20 16:36:41.706461 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/redhat-marketplace-22jh6/logs/registry-server.log 2026-03-20 16:36:41.706477 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/redhat-operators-vpvn4/ 2026-03-20 16:36:41.706493 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/redhat-operators-vpvn4/redhat-operators-vpvn4-describe 2026-03-20 16:36:41.706510 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/redhat-operators-vpvn4/logs/ 2026-03-20 16:36:41.706525 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/redhat-operators-vpvn4/logs/extract-content.log 2026-03-20 16:36:41.706541 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/redhat-operators-vpvn4/logs/extract-utilities.log 2026-03-20 16:36:41.706561 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/pods/redhat-operators-vpvn4/logs/registry-server.log.gz 2026-03-20 16:36:41.706577 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/replicaset/ 2026-03-20 16:36:41.706594 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/replicaset/marketplace-operator-79b997595.yaml 2026-03-20 16:36:41.706609 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/routes/ 2026-03-20 16:36:41.706625 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/services/ 2026-03-20 16:36:41.706641 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/services/certified-operators.yaml 2026-03-20 16:36:41.706657 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/services/community-operators.yaml 2026-03-20 16:36:41.706695 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/services/marketplace-operator-metrics.yaml 2026-03-20 16:36:41.706714 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/services/redhat-marketplace.yaml 2026-03-20 16:36:41.706730 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/services/redhat-operators.yaml 2026-03-20 16:36:41.706747 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/statefulset/ 2026-03-20 16:36:41.706763 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-marketplace/subscriptions/ 2026-03-20 16:36:41.706779 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/ 2026-03-20 16:36:41.706795 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/all_resources.log 2026-03-20 16:36:41.706816 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/events.log 2026-03-20 16:36:41.706832 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/nad.log 2026-03-20 16:36:41.706848 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/pvc.log 2026-03-20 16:36:41.706864 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/buildconfig/ 2026-03-20 16:36:41.706880 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/cert/ 2026-03-20 16:36:41.706895 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/certificaterequests/ 2026-03-20 16:36:41.706910 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/configmaps/ 2026-03-20 16:36:41.706926 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/cronjobs/ 2026-03-20 16:36:41.706941 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/daemonset/ 2026-03-20 16:36:41.706956 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/daemonset/nmstate-handler.yaml 2026-03-20 16:36:41.706971 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/deployments/ 2026-03-20 16:36:41.706986 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/deployments/nmstate-console-plugin.yaml 2026-03-20 16:36:41.707002 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/deployments/nmstate-metrics.yaml 2026-03-20 16:36:41.707018 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/deployments/nmstate-operator.yaml 2026-03-20 16:36:41.707034 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/deployments/nmstate-webhook.yaml 2026-03-20 16:36:41.707054 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/imagestream/ 2026-03-20 16:36:41.707071 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/installplans/ 2026-03-20 16:36:41.707087 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/installplans/install-6jzvn 2026-03-20 16:36:41.707102 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/issuer/ 2026-03-20 16:36:41.707118 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/jobs/ 2026-03-20 16:36:41.707133 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/poddisruptionbudgets/ 2026-03-20 16:36:41.707149 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/poddisruptionbudgets/nmstate-webhook.yaml 2026-03-20 16:36:41.707165 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/pods/ 2026-03-20 16:36:41.707180 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/pods/nmstate-console-plugin-86f58fcf4-7shs2/ 2026-03-20 16:36:41.707196 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/pods/nmstate-console-plugin-86f58fcf4-7shs2/nmstate-console-plugin-86f58fcf4-7shs2-describe 2026-03-20 16:36:41.707212 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/pods/nmstate-console-plugin-86f58fcf4-7shs2/logs/ 2026-03-20 16:36:41.707228 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/pods/nmstate-console-plugin-86f58fcf4-7shs2/logs/nmstate-console-plugin.log 2026-03-20 16:36:41.707244 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/pods/nmstate-handler-52hfr/ 2026-03-20 16:36:41.707260 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/pods/nmstate-handler-52hfr/nmstate-handler-52hfr-describe 2026-03-20 16:36:41.707280 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/pods/nmstate-handler-52hfr/logs/ 2026-03-20 16:36:41.707296 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/pods/nmstate-handler-52hfr/logs/nmstate-handler.log 2026-03-20 16:36:41.707313 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/pods/nmstate-metrics-9b8c8685d-xfs5s/ 2026-03-20 16:36:41.707329 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/pods/nmstate-metrics-9b8c8685d-xfs5s/nmstate-metrics-9b8c8685d-xfs5s-describe 2026-03-20 16:36:41.707344 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/pods/nmstate-metrics-9b8c8685d-xfs5s/logs/ 2026-03-20 16:36:41.707360 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/pods/nmstate-metrics-9b8c8685d-xfs5s/logs/kube-rbac-proxy.log 2026-03-20 16:36:41.707376 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/pods/nmstate-metrics-9b8c8685d-xfs5s/logs/nmstate-metrics.log 2026-03-20 16:36:41.707392 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/pods/nmstate-operator-796d4cfff4-s52xz/ 2026-03-20 16:36:41.707408 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/pods/nmstate-operator-796d4cfff4-s52xz/nmstate-operator-796d4cfff4-s52xz-describe 2026-03-20 16:36:41.707424 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/pods/nmstate-operator-796d4cfff4-s52xz/logs/ 2026-03-20 16:36:41.707440 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/pods/nmstate-operator-796d4cfff4-s52xz/logs/nmstate-operator.log 2026-03-20 16:36:41.707456 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/pods/nmstate-webhook-5f558f5558-xbp9m/ 2026-03-20 16:36:41.707472 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/pods/nmstate-webhook-5f558f5558-xbp9m/nmstate-webhook-5f558f5558-xbp9m-describe 2026-03-20 16:36:41.707492 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/pods/nmstate-webhook-5f558f5558-xbp9m/logs/ 2026-03-20 16:36:41.707508 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/pods/nmstate-webhook-5f558f5558-xbp9m/logs/nmstate-webhook.log 2026-03-20 16:36:41.707524 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/replicaset/ 2026-03-20 16:36:41.707540 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/replicaset/nmstate-console-plugin-86f58fcf4.yaml 2026-03-20 16:36:41.707557 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/replicaset/nmstate-metrics-9b8c8685d.yaml 2026-03-20 16:36:41.707573 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/replicaset/nmstate-operator-796d4cfff4.yaml 2026-03-20 16:36:41.707589 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/replicaset/nmstate-webhook-5f558f5558.yaml 2026-03-20 16:36:41.707605 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/routes/ 2026-03-20 16:36:41.707620 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/services/ 2026-03-20 16:36:41.707636 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/services/nmstate-console-plugin.yaml 2026-03-20 16:36:41.707652 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/services/nmstate-monitor.yaml 2026-03-20 16:36:41.707687 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/services/nmstate-webhook.yaml 2026-03-20 16:36:41.707706 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/statefulset/ 2026-03-20 16:36:41.707722 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/subscriptions/ 2026-03-20 16:36:41.707738 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-nmstate/subscriptions/kubernetes-nmstate-operator 2026-03-20 16:36:41.707760 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/ 2026-03-20 16:36:41.707777 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/all_resources.log 2026-03-20 16:36:41.707793 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/events.log 2026-03-20 16:36:41.707809 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/nad.log 2026-03-20 16:36:41.707825 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/pvc.log 2026-03-20 16:36:41.707841 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/buildconfig/ 2026-03-20 16:36:41.707857 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/cert/ 2026-03-20 16:36:41.707872 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/certificaterequests/ 2026-03-20 16:36:41.707888 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/configmaps/ 2026-03-20 16:36:41.707904 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/cronjobs/ 2026-03-20 16:36:41.707920 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/daemonset/ 2026-03-20 16:36:41.707935 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/deployments/ 2026-03-20 16:36:41.707951 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/deployments/obo-prometheus-operator-admission-webhook.yaml 2026-03-20 16:36:41.707967 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/deployments/obo-prometheus-operator.yaml 2026-03-20 16:36:41.707983 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/deployments/observability-operator.yaml 2026-03-20 16:36:41.708003 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/deployments/observability-ui-dashboards.yaml 2026-03-20 16:36:41.708019 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/deployments/perses-operator.yaml 2026-03-20 16:36:41.708035 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/imagestream/ 2026-03-20 16:36:41.708050 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/installplans/ 2026-03-20 16:36:41.708066 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/installplans/install-czkvv 2026-03-20 16:36:41.708081 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/issuer/ 2026-03-20 16:36:41.708096 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/jobs/ 2026-03-20 16:36:41.708111 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/poddisruptionbudgets/ 2026-03-20 16:36:41.708127 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/poddisruptionbudgets/obo-prometheus-operator-admission-webhook.yaml 2026-03-20 16:36:41.708142 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/pods/ 2026-03-20 16:36:41.708157 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/pods/obo-prometheus-operator-8ff7d675-zbjnw/ 2026-03-20 16:36:41.708173 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/pods/obo-prometheus-operator-8ff7d675-zbjnw/obo-prometheus-operator-8ff7d675-zbjnw-describe 2026-03-20 16:36:41.708189 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/pods/obo-prometheus-operator-8ff7d675-zbjnw/logs/ 2026-03-20 16:36:41.708204 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/pods/obo-prometheus-operator-8ff7d675-zbjnw/logs/prometheus-operator.log 2026-03-20 16:36:41.708225 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/pods/obo-prometheus-operator-admission-webhook-6b7d587c88-7c8zn/ 2026-03-20 16:36:41.708241 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/pods/obo-prometheus-operator-admission-webhook-6b7d587c88-7c8zn/obo-prometheus-operator-admission-webhook-6b7d587c88-7c8zn-describe 2026-03-20 16:36:41.708257 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/pods/obo-prometheus-operator-admission-webhook-6b7d587c88-7c8zn/logs/ 2026-03-20 16:36:41.708273 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/pods/obo-prometheus-operator-admission-webhook-6b7d587c88-7c8zn/logs/prometheus-operator-admission-webhook.log 2026-03-20 16:36:41.708289 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/pods/obo-prometheus-operator-admission-webhook-6b7d587c88-vcgbb/ 2026-03-20 16:36:41.708304 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/pods/obo-prometheus-operator-admission-webhook-6b7d587c88-vcgbb/obo-prometheus-operator-admission-webhook-6b7d587c88-vcgbb-describe 2026-03-20 16:36:41.708320 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/pods/obo-prometheus-operator-admission-webhook-6b7d587c88-vcgbb/logs/ 2026-03-20 16:36:41.708336 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/pods/obo-prometheus-operator-admission-webhook-6b7d587c88-vcgbb/logs/prometheus-operator-admission-webhook.log 2026-03-20 16:36:41.708352 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/pods/observability-operator-6dd7dd855f-bvk5s/ 2026-03-20 16:36:41.708368 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/pods/observability-operator-6dd7dd855f-bvk5s/observability-operator-6dd7dd855f-bvk5s-describe 2026-03-20 16:36:41.708397 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/pods/observability-operator-6dd7dd855f-bvk5s/logs/ 2026-03-20 16:36:41.708415 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/pods/observability-operator-6dd7dd855f-bvk5s/logs/operator.log 2026-03-20 16:36:41.708436 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/pods/observability-ui-dashboards-7f87b9b85b-6t9mn/ 2026-03-20 16:36:41.708452 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/pods/observability-ui-dashboards-7f87b9b85b-6t9mn/observability-ui-dashboards-7f87b9b85b-6t9mn-describe 2026-03-20 16:36:41.708468 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/pods/observability-ui-dashboards-7f87b9b85b-6t9mn/logs/ 2026-03-20 16:36:41.708484 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/pods/observability-ui-dashboards-7f87b9b85b-6t9mn/logs/observability-ui-dashboards.log 2026-03-20 16:36:41.708500 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/pods/perses-operator-b9bc87685-dshn8/ 2026-03-20 16:36:41.708516 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/pods/perses-operator-b9bc87685-dshn8/perses-operator-b9bc87685-dshn8-describe 2026-03-20 16:36:41.708532 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/pods/perses-operator-b9bc87685-dshn8/logs/ 2026-03-20 16:36:41.708548 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/pods/perses-operator-b9bc87685-dshn8/logs/perses-operator.log 2026-03-20 16:36:41.708564 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/replicaset/ 2026-03-20 16:36:41.708580 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/replicaset/obo-prometheus-operator-8ff7d675.yaml 2026-03-20 16:36:41.708596 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/replicaset/obo-prometheus-operator-admission-webhook-6b7d587c88.yaml 2026-03-20 16:36:41.708612 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/replicaset/observability-operator-6dd7dd855f.yaml 2026-03-20 16:36:41.708628 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/replicaset/observability-ui-dashboards-7f87b9b85b.yaml 2026-03-20 16:36:41.708648 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/replicaset/perses-operator-b9bc87685.yaml 2026-03-20 16:36:41.708677 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/routes/ 2026-03-20 16:36:41.708700 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/services/ 2026-03-20 16:36:41.708716 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/services/obo-prometheus-operator-admission-webhook-service.yaml 2026-03-20 16:36:41.708733 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/services/obo-prometheus-operator-admission-webhook.yaml 2026-03-20 16:36:41.708749 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/services/obo-prometheus-operator.yaml 2026-03-20 16:36:41.708765 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/services/observability-operator.yaml 2026-03-20 16:36:41.708781 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/services/observability-ui-dashboards.yaml 2026-03-20 16:36:41.708797 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/services/perses-operator-conversion-webhook.yaml 2026-03-20 16:36:41.708813 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/services/perses-operator-service.yaml 2026-03-20 16:36:41.708829 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/statefulset/ 2026-03-20 16:36:41.708846 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/subscriptions/ 2026-03-20 16:36:41.708864 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/subscriptions/cluster-observability-operator 2026-03-20 16:36:41.708882 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openshift-operators/subscriptions/observability-operator 2026-03-20 16:36:41.708902 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/ 2026-03-20 16:36:41.708919 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/all_resources.log 2026-03-20 16:36:41.708935 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/events.log 2026-03-20 16:36:41.708951 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/nad.log 2026-03-20 16:36:41.708967 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pvc.log 2026-03-20 16:36:41.708983 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/buildconfig/ 2026-03-20 16:36:41.708999 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/cert/ 2026-03-20 16:36:41.709015 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/cert/barbican-operator-metrics-certs.yaml 2026-03-20 16:36:41.709031 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/cert/cinder-operator-metrics-certs.yaml 2026-03-20 16:36:41.709048 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/cert/designate-operator-metrics-certs.yaml 2026-03-20 16:36:41.709064 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/cert/glance-operator-metrics-certs.yaml 2026-03-20 16:36:41.709080 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/cert/heat-operator-metrics-certs.yaml 2026-03-20 16:36:41.709097 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/cert/horizon-operator-metrics-certs.yaml 2026-03-20 16:36:41.709112 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/cert/infra-operator-metrics-certs.yaml 2026-03-20 16:36:41.709133 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/cert/infra-operator-serving-cert.yaml 2026-03-20 16:36:41.709150 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/cert/ironic-operator-metrics-certs.yaml 2026-03-20 16:36:41.709166 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/cert/keystone-operator-metrics-certs.yaml 2026-03-20 16:36:41.709181 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/cert/manila-operator-metrics-certs.yaml 2026-03-20 16:36:41.709197 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/cert/mariadb-operator-metrics-certs.yaml 2026-03-20 16:36:41.709213 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/cert/neutron-operator-metrics-certs.yaml 2026-03-20 16:36:41.709228 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/cert/nova-operator-metrics-certs.yaml 2026-03-20 16:36:41.709244 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/cert/octavia-operator-metrics-certs.yaml 2026-03-20 16:36:41.709259 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/cert/openstack-baremetal-operator-metrics-certs.yaml 2026-03-20 16:36:41.709275 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/cert/openstack-baremetal-operator-serving-cert.yaml 2026-03-20 16:36:41.709291 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/cert/openstack-operator-metrics-certs.yaml 2026-03-20 16:36:41.709307 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/cert/openstack-operator-serving-cert.yaml 2026-03-20 16:36:41.709323 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/cert/ovn-operator-metrics-certs.yaml 2026-03-20 16:36:41.709339 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/cert/placement-operator-metrics-certs.yaml 2026-03-20 16:36:41.709359 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/cert/swift-operator-metrics-certs.yaml 2026-03-20 16:36:41.709376 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/cert/telemetry-operator-metrics-certs.yaml 2026-03-20 16:36:41.709392 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/cert/test-operator-metrics-certs.yaml 2026-03-20 16:36:41.709407 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/cert/watcher-operator-metrics-certs.yaml 2026-03-20 16:36:41.709423 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/certificaterequests/ 2026-03-20 16:36:41.709438 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/certificaterequests/barbican-operator-metrics-certs-1.yaml 2026-03-20 16:36:41.709454 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/certificaterequests/cinder-operator-metrics-certs-1.yaml 2026-03-20 16:36:41.709470 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/certificaterequests/designate-operator-metrics-certs-1.yaml 2026-03-20 16:36:41.709486 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/certificaterequests/glance-operator-metrics-certs-1.yaml 2026-03-20 16:36:41.709502 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/certificaterequests/heat-operator-metrics-certs-1.yaml 2026-03-20 16:36:41.709518 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/certificaterequests/horizon-operator-metrics-certs-1.yaml 2026-03-20 16:36:41.709534 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/certificaterequests/infra-operator-metrics-certs-1.yaml 2026-03-20 16:36:41.709549 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/certificaterequests/infra-operator-serving-cert-1.yaml 2026-03-20 16:36:41.709565 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/certificaterequests/ironic-operator-metrics-certs-1.yaml 2026-03-20 16:36:41.709584 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/certificaterequests/keystone-operator-metrics-certs-1.yaml 2026-03-20 16:36:41.709600 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/certificaterequests/manila-operator-metrics-certs-1.yaml 2026-03-20 16:36:41.709615 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/certificaterequests/mariadb-operator-metrics-certs-1.yaml 2026-03-20 16:36:41.709631 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/certificaterequests/neutron-operator-metrics-certs-1.yaml 2026-03-20 16:36:41.709647 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/certificaterequests/nova-operator-metrics-certs-1.yaml 2026-03-20 16:36:41.709676 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/certificaterequests/octavia-operator-metrics-certs-1.yaml 2026-03-20 16:36:41.709698 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/certificaterequests/openstack-baremetal-operator-metrics-certs-1.yaml 2026-03-20 16:36:41.709715 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/certificaterequests/openstack-baremetal-operator-serving-cert-1.yaml 2026-03-20 16:36:41.709731 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/certificaterequests/openstack-operator-metrics-certs-1.yaml 2026-03-20 16:36:41.709747 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/certificaterequests/openstack-operator-serving-cert-1.yaml 2026-03-20 16:36:41.709763 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/certificaterequests/ovn-operator-metrics-certs-1.yaml 2026-03-20 16:36:41.709779 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/certificaterequests/placement-operator-metrics-certs-1.yaml 2026-03-20 16:36:41.709795 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/certificaterequests/swift-operator-metrics-certs-1.yaml 2026-03-20 16:36:41.709815 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/certificaterequests/telemetry-operator-metrics-certs-1.yaml 2026-03-20 16:36:41.709831 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/certificaterequests/test-operator-metrics-certs-1.yaml 2026-03-20 16:36:41.709847 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/certificaterequests/watcher-operator-metrics-certs-1.yaml 2026-03-20 16:36:41.709864 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/configmaps/ 2026-03-20 16:36:41.709880 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/cronjobs/ 2026-03-20 16:36:41.709896 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/crs/ 2026-03-20 16:36:41.709912 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/crs/openstacks.operator.openstack.org/ 2026-03-20 16:36:41.709928 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/crs/openstacks.operator.openstack.org/openstack.yaml 2026-03-20 16:36:41.709944 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/daemonset/ 2026-03-20 16:36:41.709959 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/deployments/ 2026-03-20 16:36:41.709975 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/deployments/barbican-operator-controller-manager.yaml 2026-03-20 16:36:41.709991 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/deployments/cinder-operator-controller-manager.yaml 2026-03-20 16:36:41.710007 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/deployments/designate-operator-controller-manager.yaml 2026-03-20 16:36:41.710022 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/deployments/glance-operator-controller-manager.yaml 2026-03-20 16:36:41.710044 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/deployments/heat-operator-controller-manager.yaml 2026-03-20 16:36:41.710060 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/deployments/horizon-operator-controller-manager.yaml 2026-03-20 16:36:41.710076 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/deployments/infra-operator-controller-manager.yaml 2026-03-20 16:36:41.710092 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/deployments/ironic-operator-controller-manager.yaml 2026-03-20 16:36:41.710107 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/deployments/keystone-operator-controller-manager.yaml 2026-03-20 16:36:41.710124 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/deployments/manila-operator-controller-manager.yaml 2026-03-20 16:36:41.710140 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/deployments/mariadb-operator-controller-manager.yaml 2026-03-20 16:36:41.710156 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/deployments/neutron-operator-controller-manager.yaml 2026-03-20 16:36:41.710172 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/deployments/nova-operator-controller-manager.yaml 2026-03-20 16:36:41.710188 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/deployments/octavia-operator-controller-manager.yaml 2026-03-20 16:36:41.710203 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/deployments/openstack-baremetal-operator-controller-manager.yaml 2026-03-20 16:36:41.710219 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/deployments/openstack-operator-controller-init.yaml 2026-03-20 16:36:41.710235 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/deployments/openstack-operator-controller-manager.yaml 2026-03-20 16:36:41.710256 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/deployments/ovn-operator-controller-manager.yaml 2026-03-20 16:36:41.710272 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/deployments/placement-operator-controller-manager.yaml 2026-03-20 16:36:41.710288 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/deployments/rabbitmq-cluster-operator-manager.yaml 2026-03-20 16:36:41.710304 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/deployments/swift-operator-controller-manager.yaml 2026-03-20 16:36:41.710319 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/deployments/telemetry-operator-controller-manager.yaml 2026-03-20 16:36:41.710335 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/deployments/test-operator-controller-manager.yaml 2026-03-20 16:36:41.710350 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/deployments/watcher-operator-controller-manager.yaml 2026-03-20 16:36:41.710366 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/imagestream/ 2026-03-20 16:36:41.710381 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/installplans/ 2026-03-20 16:36:41.710397 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/installplans/install-225v4 2026-03-20 16:36:41.710413 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/installplans/install-k7phs 2026-03-20 16:36:41.710428 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/issuer/ 2026-03-20 16:36:41.710444 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/issuer/barbican-operator-selfsigned-issuer.yaml 2026-03-20 16:36:41.710459 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/issuer/cinder-operator-selfsigned-issuer.yaml 2026-03-20 16:36:41.710480 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/issuer/designate-operator-selfsigned-issuer.yaml 2026-03-20 16:36:41.710496 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/issuer/glance-operator-selfsigned-issuer.yaml 2026-03-20 16:36:41.710512 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/issuer/heat-operator-selfsigned-issuer.yaml 2026-03-20 16:36:41.710527 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/issuer/horizon-operator-selfsigned-issuer.yaml 2026-03-20 16:36:41.710543 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/issuer/infra-operator-selfsigned-issuer.yaml 2026-03-20 16:36:41.710558 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/issuer/ironic-operator-selfsigned-issuer.yaml 2026-03-20 16:36:41.710573 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/issuer/keystone-operator-selfsigned-issuer.yaml 2026-03-20 16:36:41.710589 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/issuer/manila-operator-selfsigned-issuer.yaml 2026-03-20 16:36:41.710605 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/issuer/mariadb-operator-selfsigned-issuer.yaml 2026-03-20 16:36:41.710621 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/issuer/neutron-operator-selfsigned-issuer.yaml 2026-03-20 16:36:41.710636 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/issuer/nova-operator-selfsigned-issuer.yaml 2026-03-20 16:36:41.710652 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/issuer/octavia-operator-selfsigned-issuer.yaml 2026-03-20 16:36:41.710687 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/issuer/openstack-baremetal-operator-selfsigned-issuer.yaml 2026-03-20 16:36:41.710710 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/issuer/openstack-operator-selfsigned-issuer.yaml 2026-03-20 16:36:41.710726 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/issuer/ovn-operator-selfsigned-issuer.yaml 2026-03-20 16:36:41.710743 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/issuer/placement-operator-selfsigned-issuer.yaml 2026-03-20 16:36:41.710758 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/issuer/rabbitmq-cluster-operator-selfsigned-issuer.yaml 2026-03-20 16:36:41.710774 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/issuer/swift-operator-selfsigned-issuer.yaml 2026-03-20 16:36:41.710791 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/issuer/telemetry-operator-selfsigned-issuer.yaml 2026-03-20 16:36:41.710807 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/issuer/test-operator-selfsigned-issuer.yaml 2026-03-20 16:36:41.710823 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/issuer/watcher-operator-selfsigned-issuer.yaml 2026-03-20 16:36:41.710839 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/jobs/ 2026-03-20 16:36:41.710855 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/jobs/6e5835fdb519dbfa6963d219a5ec71b0d73708b5df93d36b8aba1618ba3eca4.yaml 2026-03-20 16:36:41.710871 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/jobs/8def7df2b499ac483c55b958477ae70d30de43fc359ef64c96f9aee971080d7.yaml 2026-03-20 16:36:41.710887 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/poddisruptionbudgets/ 2026-03-20 16:36:41.710902 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/ 2026-03-20 16:36:41.710918 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/6e5835fdb519dbfa6963d219a5ec71b0d73708b5df93d36b8aba1618batvcbb/ 2026-03-20 16:36:41.710938 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/6e5835fdb519dbfa6963d219a5ec71b0d73708b5df93d36b8aba1618batvcbb/6e5835fdb519dbfa6963d219a5ec71b0d73708b5df93d36b8aba1618batvcbb-describe 2026-03-20 16:36:41.710955 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/6e5835fdb519dbfa6963d219a5ec71b0d73708b5df93d36b8aba1618batvcbb/logs/ 2026-03-20 16:36:41.710971 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/6e5835fdb519dbfa6963d219a5ec71b0d73708b5df93d36b8aba1618batvcbb/logs/extract.log 2026-03-20 16:36:41.710987 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/6e5835fdb519dbfa6963d219a5ec71b0d73708b5df93d36b8aba1618batvcbb/logs/pull.log 2026-03-20 16:36:41.711003 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/6e5835fdb519dbfa6963d219a5ec71b0d73708b5df93d36b8aba1618batvcbb/logs/util.log 2026-03-20 16:36:41.711019 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/8def7df2b499ac483c55b958477ae70d30de43fc359ef64c96f9aee971khqb2/ 2026-03-20 16:36:41.711035 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/8def7df2b499ac483c55b958477ae70d30de43fc359ef64c96f9aee971khqb2/8def7df2b499ac483c55b958477ae70d30de43fc359ef64c96f9aee971khqb2-describe 2026-03-20 16:36:41.711051 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/8def7df2b499ac483c55b958477ae70d30de43fc359ef64c96f9aee971khqb2/logs/ 2026-03-20 16:36:41.711067 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/8def7df2b499ac483c55b958477ae70d30de43fc359ef64c96f9aee971khqb2/logs/extract.log 2026-03-20 16:36:41.711083 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/8def7df2b499ac483c55b958477ae70d30de43fc359ef64c96f9aee971khqb2/logs/pull.log 2026-03-20 16:36:41.711099 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/8def7df2b499ac483c55b958477ae70d30de43fc359ef64c96f9aee971khqb2/logs/util.log 2026-03-20 16:36:41.711114 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/barbican-operator-controller-manager-59bc569d95-2l2s5/ 2026-03-20 16:36:41.711136 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/barbican-operator-controller-manager-59bc569d95-2l2s5/barbican-operator-controller-manager-59bc569d95-2l2s5-describe 2026-03-20 16:36:41.711153 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/barbican-operator-controller-manager-59bc569d95-2l2s5/logs/ 2026-03-20 16:36:41.711169 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/barbican-operator-controller-manager-59bc569d95-2l2s5/logs/manager.log 2026-03-20 16:36:41.711184 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/cinder-operator-controller-manager-8d58dc466-q88gq/ 2026-03-20 16:36:41.711200 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/cinder-operator-controller-manager-8d58dc466-q88gq/cinder-operator-controller-manager-8d58dc466-q88gq-describe 2026-03-20 16:36:41.711216 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/cinder-operator-controller-manager-8d58dc466-q88gq/logs/ 2026-03-20 16:36:41.711232 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/cinder-operator-controller-manager-8d58dc466-q88gq/logs/manager.log.gz 2026-03-20 16:36:41.711248 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/designate-operator-controller-manager-588d4d986b-lzq5s/ 2026-03-20 16:36:41.711264 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/designate-operator-controller-manager-588d4d986b-lzq5s/designate-operator-controller-manager-588d4d986b-lzq5s-describe 2026-03-20 16:36:41.711280 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/designate-operator-controller-manager-588d4d986b-lzq5s/logs/ 2026-03-20 16:36:41.711296 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/designate-operator-controller-manager-588d4d986b-lzq5s/logs/manager.log 2026-03-20 16:36:41.711312 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/glance-operator-controller-manager-79df6bcc97-z8zt6/ 2026-03-20 16:36:41.711332 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/glance-operator-controller-manager-79df6bcc97-z8zt6/glance-operator-controller-manager-79df6bcc97-z8zt6-describe 2026-03-20 16:36:41.711349 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/glance-operator-controller-manager-79df6bcc97-z8zt6/logs/ 2026-03-20 16:36:41.711365 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/glance-operator-controller-manager-79df6bcc97-z8zt6/logs/manager.log 2026-03-20 16:36:41.711381 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/heat-operator-controller-manager-67dd5f86f5-mqm85/ 2026-03-20 16:36:41.711397 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/heat-operator-controller-manager-67dd5f86f5-mqm85/heat-operator-controller-manager-67dd5f86f5-mqm85-describe 2026-03-20 16:36:41.711413 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/heat-operator-controller-manager-67dd5f86f5-mqm85/logs/ 2026-03-20 16:36:41.711429 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/heat-operator-controller-manager-67dd5f86f5-mqm85/logs/manager.log 2026-03-20 16:36:41.711445 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/horizon-operator-controller-manager-8464cc45fb-v8t7w/ 2026-03-20 16:36:41.711461 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/horizon-operator-controller-manager-8464cc45fb-v8t7w/horizon-operator-controller-manager-8464cc45fb-v8t7w-describe 2026-03-20 16:36:41.711477 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/horizon-operator-controller-manager-8464cc45fb-v8t7w/logs/ 2026-03-20 16:36:41.711493 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/horizon-operator-controller-manager-8464cc45fb-v8t7w/logs/manager.log 2026-03-20 16:36:41.711509 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/infra-operator-controller-manager-7b9c774f96-qzktk/ 2026-03-20 16:36:41.711530 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/infra-operator-controller-manager-7b9c774f96-qzktk/infra-operator-controller-manager-7b9c774f96-qzktk-describe 2026-03-20 16:36:41.711546 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/infra-operator-controller-manager-7b9c774f96-qzktk/logs/ 2026-03-20 16:36:41.711562 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/infra-operator-controller-manager-7b9c774f96-qzktk/logs/manager.log 2026-03-20 16:36:41.711578 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/ironic-operator-controller-manager-6f787dddc9-r7v2l/ 2026-03-20 16:36:41.711594 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/ironic-operator-controller-manager-6f787dddc9-r7v2l/ironic-operator-controller-manager-6f787dddc9-r7v2l-describe 2026-03-20 16:36:41.711610 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/ironic-operator-controller-manager-6f787dddc9-r7v2l/logs/ 2026-03-20 16:36:41.711626 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/ironic-operator-controller-manager-6f787dddc9-r7v2l/logs/manager.log 2026-03-20 16:36:41.711643 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/keystone-operator-controller-manager-768b96df4c-kvw9r/ 2026-03-20 16:36:41.711659 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/keystone-operator-controller-manager-768b96df4c-kvw9r/keystone-operator-controller-manager-768b96df4c-kvw9r-describe 2026-03-20 16:36:41.711695 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/keystone-operator-controller-manager-768b96df4c-kvw9r/logs/ 2026-03-20 16:36:41.711713 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/keystone-operator-controller-manager-768b96df4c-kvw9r/logs/manager.log 2026-03-20 16:36:41.711729 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/manila-operator-controller-manager-55f864c847-6xnn6/ 2026-03-20 16:36:41.711749 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/manila-operator-controller-manager-55f864c847-6xnn6/manila-operator-controller-manager-55f864c847-6xnn6-describe 2026-03-20 16:36:41.711765 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/manila-operator-controller-manager-55f864c847-6xnn6/logs/ 2026-03-20 16:36:41.711781 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/manila-operator-controller-manager-55f864c847-6xnn6/logs/manager.log 2026-03-20 16:36:41.711798 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/mariadb-operator-controller-manager-67ccfc9778-cff9z/ 2026-03-20 16:36:41.711814 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/mariadb-operator-controller-manager-67ccfc9778-cff9z/mariadb-operator-controller-manager-67ccfc9778-cff9z-describe 2026-03-20 16:36:41.711830 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/mariadb-operator-controller-manager-67ccfc9778-cff9z/logs/ 2026-03-20 16:36:41.711846 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/mariadb-operator-controller-manager-67ccfc9778-cff9z/logs/manager.log 2026-03-20 16:36:41.711861 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/neutron-operator-controller-manager-767865f676-fstk4/ 2026-03-20 16:36:41.711877 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/neutron-operator-controller-manager-767865f676-fstk4/neutron-operator-controller-manager-767865f676-fstk4-describe 2026-03-20 16:36:41.711893 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/neutron-operator-controller-manager-767865f676-fstk4/logs/ 2026-03-20 16:36:41.711910 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/neutron-operator-controller-manager-767865f676-fstk4/logs/manager.log 2026-03-20 16:36:41.711926 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/nova-operator-controller-manager-5d488d59fb-vj727/ 2026-03-20 16:36:41.711941 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/nova-operator-controller-manager-5d488d59fb-vj727/nova-operator-controller-manager-5d488d59fb-vj727-describe 2026-03-20 16:36:41.711962 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/nova-operator-controller-manager-5d488d59fb-vj727/logs/ 2026-03-20 16:36:41.711979 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/nova-operator-controller-manager-5d488d59fb-vj727/logs/manager.log 2026-03-20 16:36:41.711995 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/octavia-operator-controller-manager-5b9f45d989-rncs2/ 2026-03-20 16:36:41.712010 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/octavia-operator-controller-manager-5b9f45d989-rncs2/octavia-operator-controller-manager-5b9f45d989-rncs2-describe 2026-03-20 16:36:41.712026 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/octavia-operator-controller-manager-5b9f45d989-rncs2/logs/ 2026-03-20 16:36:41.712042 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/octavia-operator-controller-manager-5b9f45d989-rncs2/logs/manager.log 2026-03-20 16:36:41.712058 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/openstack-baremetal-operator-controller-manager-89d64c458-ntg7z/ 2026-03-20 16:36:41.712073 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/openstack-baremetal-operator-controller-manager-89d64c458-ntg7z/openstack-baremetal-operator-controller-manager-89d64c458-ntg7z-describe 2026-03-20 16:36:41.712089 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/openstack-baremetal-operator-controller-manager-89d64c458-ntg7z/logs/ 2026-03-20 16:36:41.712105 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/openstack-baremetal-operator-controller-manager-89d64c458-ntg7z/logs/manager.log 2026-03-20 16:36:41.712122 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/openstack-operator-controller-manager-6f58c59cbb-66v95/ 2026-03-20 16:36:41.712138 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/openstack-operator-controller-manager-6f58c59cbb-66v95/openstack-operator-controller-manager-6f58c59cbb-66v95-describe 2026-03-20 16:36:41.712158 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/openstack-operator-controller-manager-6f58c59cbb-66v95/logs/ 2026-03-20 16:36:41.712174 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/openstack-operator-controller-manager-6f58c59cbb-66v95/logs/manager.log.gz 2026-03-20 16:36:41.712191 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/openstack-operator-index-46f5x/ 2026-03-20 16:36:41.712207 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/openstack-operator-index-46f5x/openstack-operator-index-46f5x-describe 2026-03-20 16:36:41.712223 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/openstack-operator-index-46f5x/logs/ 2026-03-20 16:36:41.712239 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/openstack-operator-index-46f5x/logs/registry-server.log 2026-03-20 16:36:41.712255 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/ovn-operator-controller-manager-884679f54-xvk2s/ 2026-03-20 16:36:41.712271 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/ovn-operator-controller-manager-884679f54-xvk2s/ovn-operator-controller-manager-884679f54-xvk2s-describe 2026-03-20 16:36:41.712287 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/ovn-operator-controller-manager-884679f54-xvk2s/logs/ 2026-03-20 16:36:41.712302 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/ovn-operator-controller-manager-884679f54-xvk2s/logs/manager.log 2026-03-20 16:36:41.712318 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/placement-operator-controller-manager-5784578c99-gpxm6/ 2026-03-20 16:36:41.712334 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/placement-operator-controller-manager-5784578c99-gpxm6/placement-operator-controller-manager-5784578c99-gpxm6-describe 2026-03-20 16:36:41.712353 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/placement-operator-controller-manager-5784578c99-gpxm6/logs/ 2026-03-20 16:36:41.712370 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/placement-operator-controller-manager-5784578c99-gpxm6/logs/manager.log 2026-03-20 16:36:41.712386 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/rabbitmq-cluster-operator-manager-668c99d594-jmwcp/ 2026-03-20 16:36:41.712417 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/rabbitmq-cluster-operator-manager-668c99d594-jmwcp/rabbitmq-cluster-operator-manager-668c99d594-jmwcp-describe 2026-03-20 16:36:41.712434 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/rabbitmq-cluster-operator-manager-668c99d594-jmwcp/logs/ 2026-03-20 16:36:41.712451 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/rabbitmq-cluster-operator-manager-668c99d594-jmwcp/logs/operator.log 2026-03-20 16:36:41.712466 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/swift-operator-controller-manager-c674c5965-nqn84/ 2026-03-20 16:36:41.712482 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/swift-operator-controller-manager-c674c5965-nqn84/swift-operator-controller-manager-c674c5965-nqn84-describe 2026-03-20 16:36:41.712498 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/swift-operator-controller-manager-c674c5965-nqn84/logs/ 2026-03-20 16:36:41.712513 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/swift-operator-controller-manager-c674c5965-nqn84/logs/manager.log 2026-03-20 16:36:41.712529 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/telemetry-operator-controller-manager-d6b694c5-2xj66/ 2026-03-20 16:36:41.712544 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/telemetry-operator-controller-manager-d6b694c5-2xj66/telemetry-operator-controller-manager-d6b694c5-2xj66-describe 2026-03-20 16:36:41.712563 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/telemetry-operator-controller-manager-d6b694c5-2xj66/logs/ 2026-03-20 16:36:41.712579 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/telemetry-operator-controller-manager-d6b694c5-2xj66/logs/manager.log.gz 2026-03-20 16:36:41.712595 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/test-operator-controller-manager-5c5cb9c4d7-5rxh5/ 2026-03-20 16:36:41.712611 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/test-operator-controller-manager-5c5cb9c4d7-5rxh5/test-operator-controller-manager-5c5cb9c4d7-5rxh5-describe 2026-03-20 16:36:41.712626 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/test-operator-controller-manager-5c5cb9c4d7-5rxh5/logs/ 2026-03-20 16:36:41.712642 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/test-operator-controller-manager-5c5cb9c4d7-5rxh5/logs/manager.log 2026-03-20 16:36:41.712657 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/watcher-operator-controller-manager-77c9f8cb5b-z626z/ 2026-03-20 16:36:41.712694 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/watcher-operator-controller-manager-77c9f8cb5b-z626z/watcher-operator-controller-manager-77c9f8cb5b-z626z-describe 2026-03-20 16:36:41.712713 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/watcher-operator-controller-manager-77c9f8cb5b-z626z/logs/ 2026-03-20 16:36:41.712729 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/watcher-operator-controller-manager-77c9f8cb5b-z626z/logs/manager.log.gz 2026-03-20 16:36:41.712745 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/watcher-operator-index-5c49m/ 2026-03-20 16:36:41.712761 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/watcher-operator-index-5c49m/watcher-operator-index-5c49m-describe 2026-03-20 16:36:41.712777 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/watcher-operator-index-5c49m/logs/ 2026-03-20 16:36:41.712798 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/pods/watcher-operator-index-5c49m/logs/registry-server.log 2026-03-20 16:36:41.712814 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/replicaset/ 2026-03-20 16:36:41.712830 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/replicaset/barbican-operator-controller-manager-59bc569d95.yaml 2026-03-20 16:36:41.712847 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/replicaset/cinder-operator-controller-manager-8d58dc466.yaml 2026-03-20 16:36:41.712865 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/replicaset/designate-operator-controller-manager-588d4d986b.yaml 2026-03-20 16:36:41.712883 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/replicaset/glance-operator-controller-manager-79df6bcc97.yaml 2026-03-20 16:36:41.712899 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/replicaset/heat-operator-controller-manager-67dd5f86f5.yaml 2026-03-20 16:36:41.712914 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/replicaset/horizon-operator-controller-manager-8464cc45fb.yaml 2026-03-20 16:36:41.712930 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/replicaset/infra-operator-controller-manager-7b9c774f96.yaml 2026-03-20 16:36:41.712946 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/replicaset/ironic-operator-controller-manager-6f787dddc9.yaml 2026-03-20 16:36:41.712961 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/replicaset/keystone-operator-controller-manager-768b96df4c.yaml 2026-03-20 16:36:41.712977 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/replicaset/manila-operator-controller-manager-55f864c847.yaml 2026-03-20 16:36:41.712993 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/replicaset/mariadb-operator-controller-manager-67ccfc9778.yaml 2026-03-20 16:36:41.713012 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/replicaset/neutron-operator-controller-manager-767865f676.yaml 2026-03-20 16:36:41.713028 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/replicaset/nova-operator-controller-manager-5d488d59fb.yaml 2026-03-20 16:36:41.713043 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/replicaset/octavia-operator-controller-manager-5b9f45d989.yaml 2026-03-20 16:36:41.713059 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/replicaset/openstack-baremetal-operator-controller-manager-89d64c458.yaml 2026-03-20 16:36:41.713074 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/replicaset/openstack-operator-controller-init-646f48576b.yaml 2026-03-20 16:36:41.713089 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/replicaset/openstack-operator-controller-manager-6f58c59cbb.yaml 2026-03-20 16:36:41.713104 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/replicaset/ovn-operator-controller-manager-884679f54.yaml 2026-03-20 16:36:41.713120 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/replicaset/placement-operator-controller-manager-5784578c99.yaml 2026-03-20 16:36:41.713135 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/replicaset/rabbitmq-cluster-operator-manager-668c99d594.yaml 2026-03-20 16:36:41.713151 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/replicaset/swift-operator-controller-manager-c674c5965.yaml 2026-03-20 16:36:41.713167 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/replicaset/telemetry-operator-controller-manager-d6b694c5.yaml 2026-03-20 16:36:41.713182 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/replicaset/test-operator-controller-manager-5c5cb9c4d7.yaml 2026-03-20 16:36:41.713198 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/replicaset/watcher-operator-controller-manager-77c9f8cb5b.yaml 2026-03-20 16:36:41.713218 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/routes/ 2026-03-20 16:36:41.713234 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/secrets/ 2026-03-20 16:36:41.713249 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/secrets/barbican/ 2026-03-20 16:36:41.713265 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/secrets/barbican/barbican-operator-metrics-server-cert.yaml 2026-03-20 16:36:41.713281 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/secrets/cinder/ 2026-03-20 16:36:41.713297 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/secrets/cinder/cinder-operator-metrics-server-cert.yaml 2026-03-20 16:36:41.713313 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/secrets/designate/ 2026-03-20 16:36:41.713329 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/secrets/designate/designate-operator-metrics-server-cert.yaml 2026-03-20 16:36:41.713344 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/secrets/glance/ 2026-03-20 16:36:41.713360 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/secrets/glance/glance-operator-metrics-server-cert.yaml 2026-03-20 16:36:41.713375 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/secrets/heat/ 2026-03-20 16:36:41.713391 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/secrets/heat/heat-operator-metrics-server-cert.yaml 2026-03-20 16:36:41.713407 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/secrets/horizon/ 2026-03-20 16:36:41.713422 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/secrets/horizon/horizon-operator-metrics-server-cert.yaml 2026-03-20 16:36:41.713443 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/secrets/ironic/ 2026-03-20 16:36:41.713459 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/secrets/ironic/ironic-operator-metrics-server-cert.yaml 2026-03-20 16:36:41.713475 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/secrets/keystone/ 2026-03-20 16:36:41.713491 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/secrets/keystone/keystone-operator-metrics-server-cert.yaml 2026-03-20 16:36:41.713507 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/secrets/manila/ 2026-03-20 16:36:41.713523 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/secrets/manila/manila-operator-metrics-server-cert.yaml 2026-03-20 16:36:41.713539 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/secrets/mariadb/ 2026-03-20 16:36:41.713555 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/secrets/mariadb/mariadb-operator-metrics-server-cert.yaml 2026-03-20 16:36:41.713571 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/secrets/neutron/ 2026-03-20 16:36:41.713587 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/secrets/neutron/neutron-operator-metrics-server-cert.yaml 2026-03-20 16:36:41.713602 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/secrets/nova/ 2026-03-20 16:36:41.713618 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/secrets/nova/nova-operator-metrics-server-cert.yaml 2026-03-20 16:36:41.713634 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/secrets/octavia/ 2026-03-20 16:36:41.713649 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/secrets/octavia/octavia-operator-metrics-server-cert.yaml 2026-03-20 16:36:41.713685 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/secrets/ovn/ 2026-03-20 16:36:41.713707 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/secrets/ovn/ovn-operator-metrics-server-cert.yaml 2026-03-20 16:36:41.713723 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/secrets/placement/ 2026-03-20 16:36:41.713739 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/secrets/placement/placement-operator-metrics-server-cert.yaml 2026-03-20 16:36:41.713755 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/secrets/swift/ 2026-03-20 16:36:41.713771 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/secrets/swift/swift-operator-metrics-server-cert.yaml 2026-03-20 16:36:41.713787 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/secrets/telemetry/ 2026-03-20 16:36:41.713802 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/secrets/telemetry/telemetry-operator-metrics-server-cert.yaml 2026-03-20 16:36:41.713818 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/secrets/watcher/ 2026-03-20 16:36:41.713834 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/secrets/watcher/watcher-operator-controller-manager-service-cert.yaml 2026-03-20 16:36:41.713849 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/secrets/watcher/watcher-operator-metrics-server-cert.yaml 2026-03-20 16:36:41.713865 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/services/ 2026-03-20 16:36:41.713881 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/services/barbican-operator-controller-manager-metrics-service.yaml 2026-03-20 16:36:41.713896 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/services/cinder-operator-controller-manager-metrics-service.yaml 2026-03-20 16:36:41.713917 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/services/designate-operator-controller-manager-metrics-service.yaml 2026-03-20 16:36:41.713933 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/services/glance-operator-controller-manager-metrics-service.yaml 2026-03-20 16:36:41.713949 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/services/heat-operator-controller-manager-metrics-service.yaml 2026-03-20 16:36:41.713965 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/services/horizon-operator-controller-manager-metrics-service.yaml 2026-03-20 16:36:41.713980 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/services/infra-operator-controller-manager-metrics-service.yaml 2026-03-20 16:36:41.713996 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/services/infra-operator-webhook-service.yaml 2026-03-20 16:36:41.714012 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/services/ironic-operator-controller-manager-metrics-service.yaml 2026-03-20 16:36:41.714028 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/services/keystone-operator-controller-manager-metrics-service.yaml 2026-03-20 16:36:41.714044 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/services/manila-operator-controller-manager-metrics-service.yaml 2026-03-20 16:36:41.714060 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/services/mariadb-operator-controller-manager-metrics-service.yaml 2026-03-20 16:36:41.714076 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/services/neutron-operator-controller-manager-metrics-service.yaml 2026-03-20 16:36:41.714092 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/services/nova-operator-controller-manager-metrics-service.yaml 2026-03-20 16:36:41.714108 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/services/octavia-operator-controller-manager-metrics-service.yaml 2026-03-20 16:36:41.714128 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/services/openstack-baremetal-operator-controller-manager-metrics-service.yaml 2026-03-20 16:36:41.714145 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/services/openstack-baremetal-operator-webhook-service.yaml 2026-03-20 16:36:41.714161 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/services/openstack-operator-controller-manager-metrics-service.yaml 2026-03-20 16:36:41.714177 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/services/openstack-operator-index.yaml 2026-03-20 16:36:41.714193 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/services/openstack-operator-webhook-service.yaml 2026-03-20 16:36:41.714209 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/services/ovn-operator-controller-manager-metrics-service.yaml 2026-03-20 16:36:41.714225 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/services/placement-operator-controller-manager-metrics-service.yaml 2026-03-20 16:36:41.714241 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/services/rabbitmq-cluster-operator-controller-manager-metrics-service.yaml 2026-03-20 16:36:41.714257 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/services/swift-operator-controller-manager-metrics-service.yaml 2026-03-20 16:36:41.714272 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/services/telemetry-operator-controller-manager-metrics-service.yaml 2026-03-20 16:36:41.714350 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/services/test-operator-controller-manager-metrics-service.yaml 2026-03-20 16:36:41.714377 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/services/watcher-operator-controller-manager-metrics-service.yaml 2026-03-20 16:36:41.714403 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/services/watcher-operator-controller-manager-service.yaml 2026-03-20 16:36:41.714421 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/services/watcher-operator-index.yaml 2026-03-20 16:36:41.714438 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/services/watcher-operator-webhook-service.yaml 2026-03-20 16:36:41.714455 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/statefulset/ 2026-03-20 16:36:41.714472 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/subscriptions/ 2026-03-20 16:36:41.714490 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/subscriptions/openstack-operator 2026-03-20 16:36:41.714507 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack-operators/subscriptions/watcher-operator 2026-03-20 16:36:41.714524 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack/ 2026-03-20 16:36:41.714541 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack/all_resources.log 2026-03-20 16:36:41.714557 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack/events.log 2026-03-20 16:36:41.714574 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack/nad.log 2026-03-20 16:36:41.714590 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack/pvc.log 2026-03-20 16:36:41.714606 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack/buildconfig/ 2026-03-20 16:36:41.714622 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack/cert/ 2026-03-20 16:36:41.714638 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack/certificaterequests/ 2026-03-20 16:36:41.714658 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack/configmaps/ 2026-03-20 16:36:41.714698 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack/cronjobs/ 2026-03-20 16:36:41.714717 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack/crs/ 2026-03-20 16:36:41.714733 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/ 2026-03-20 16:36:41.714749 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/ctlplane.yaml 2026-03-20 16:36:41.714766 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/datacentre.yaml 2026-03-20 16:36:41.714783 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/designate.yaml 2026-03-20 16:36:41.714799 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/designateext.yaml 2026-03-20 16:36:41.714815 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/internalapi.yaml 2026-03-20 16:36:41.714831 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/octavia.yaml 2026-03-20 16:36:41.714847 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/storage.yaml 2026-03-20 16:36:41.714863 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/storagemgmt.yaml 2026-03-20 16:36:41.714879 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack/crs/network-attachment-definitions.k8s.cni.cncf.io/tenant.yaml 2026-03-20 16:36:41.714895 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack/daemonset/ 2026-03-20 16:36:41.714916 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack/deployments/ 2026-03-20 16:36:41.714932 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack/imagestream/ 2026-03-20 16:36:41.714948 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack/installplans/ 2026-03-20 16:36:41.714964 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack/issuer/ 2026-03-20 16:36:41.714979 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack/jobs/ 2026-03-20 16:36:41.714995 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack/poddisruptionbudgets/ 2026-03-20 16:36:41.715011 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack/pods/ 2026-03-20 16:36:41.715027 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack/pods/-describe 2026-03-20 16:36:41.715043 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack/pods/logs/ 2026-03-20 16:36:41.715059 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack/replicaset/ 2026-03-20 16:36:41.715075 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack/routes/ 2026-03-20 16:36:41.715090 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack/services/ 2026-03-20 16:36:41.715106 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack/statefulset/ 2026-03-20 16:36:41.715121 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/openstack/subscriptions/ 2026-03-20 16:36:41.715137 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/ 2026-03-20 16:36:41.715152 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/all_resources.log 2026-03-20 16:36:41.715173 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/events.log 2026-03-20 16:36:41.715188 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/nad.log 2026-03-20 16:36:41.715203 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pvc.log 2026-03-20 16:36:41.715219 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/buildconfig/ 2026-03-20 16:36:41.715234 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/cert/ 2026-03-20 16:36:41.715249 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/cert/ceilometer-internal-svc.yaml 2026-03-20 16:36:41.715264 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/cert/cinder-internal-svc.yaml 2026-03-20 16:36:41.715280 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/cert/galera-openstack-svc.yaml 2026-03-20 16:36:41.715296 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/cert/keystone-internal-svc.yaml 2026-03-20 16:36:41.715312 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/cert/keystone-public-route.yaml 2026-03-20 16:36:41.715327 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/cert/keystone-public-svc.yaml 2026-03-20 16:36:41.715343 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/cert/kube-state-metrics-svc.yaml 2026-03-20 16:36:41.715358 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/cert/memcached-mtls.yaml 2026-03-20 16:36:41.715374 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/cert/memcached-svc.yaml 2026-03-20 16:36:41.715389 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/cert/metric-storage-prometheus-svc.yaml 2026-03-20 16:36:41.715412 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/cert/rabbitmq-notifications-svc.yaml 2026-03-20 16:36:41.715428 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/cert/rabbitmq-svc.yaml 2026-03-20 16:36:41.715444 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/cert/rootca-internal.yaml 2026-03-20 16:36:41.715459 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/cert/rootca-libvirt.yaml 2026-03-20 16:36:41.715475 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/cert/rootca-ovn.yaml 2026-03-20 16:36:41.715490 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/cert/rootca-public.yaml 2026-03-20 16:36:41.715505 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/certificaterequests/ 2026-03-20 16:36:41.715521 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/certificaterequests/ceilometer-internal-svc-1.yaml 2026-03-20 16:36:41.715536 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/certificaterequests/cinder-internal-svc-1.yaml 2026-03-20 16:36:41.715552 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/certificaterequests/galera-openstack-svc-1.yaml 2026-03-20 16:36:41.715568 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/certificaterequests/keystone-internal-svc-1.yaml 2026-03-20 16:36:41.715584 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/certificaterequests/keystone-public-route-1.yaml 2026-03-20 16:36:41.715600 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/certificaterequests/keystone-public-svc-1.yaml 2026-03-20 16:36:41.715616 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/certificaterequests/kube-state-metrics-svc-1.yaml 2026-03-20 16:36:41.715637 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/certificaterequests/memcached-mtls-1.yaml 2026-03-20 16:36:41.715654 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/certificaterequests/memcached-svc-1.yaml 2026-03-20 16:36:41.715690 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/certificaterequests/metric-storage-prometheus-svc-1.yaml 2026-03-20 16:36:41.715708 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/certificaterequests/rabbitmq-notifications-svc-1.yaml 2026-03-20 16:36:41.715724 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/certificaterequests/rabbitmq-svc-1.yaml 2026-03-20 16:36:41.715740 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/certificaterequests/rootca-internal-1.yaml 2026-03-20 16:36:41.715755 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/certificaterequests/rootca-libvirt-1.yaml 2026-03-20 16:36:41.715771 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/certificaterequests/rootca-ovn-1.yaml 2026-03-20 16:36:41.715787 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/certificaterequests/rootca-public-1.yaml 2026-03-20 16:36:41.715802 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/configmaps/ 2026-03-20 16:36:41.715818 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/configmaps/appcred-watcher-pre.yaml 2026-03-20 16:36:41.715833 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/configmaps/infra-instanceha-config.yaml 2026-03-20 16:36:41.715849 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/configmaps/prometheus-metric-storage-rulefiles-0.yaml 2026-03-20 16:36:41.715872 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/configmaps/rabbitmq-config-data.yaml 2026-03-20 16:36:41.715891 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/configmaps/rabbitmq-notifications-config-data.yaml 2026-03-20 16:36:41.715907 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/configmaps/rabbitmq-notifications-plugins-conf.yaml 2026-03-20 16:36:41.715924 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/configmaps/rabbitmq-notifications-server-conf.yaml 2026-03-20 16:36:41.715940 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/configmaps/rabbitmq-plugins-conf.yaml 2026-03-20 16:36:41.715956 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/configmaps/rabbitmq-server-conf.yaml 2026-03-20 16:36:41.715972 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/cronjobs/ 2026-03-20 16:36:41.715988 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/cronjobs/keystone-cron.yaml 2026-03-20 16:36:41.716003 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/ 2026-03-20 16:36:41.716019 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/alertmanagers.monitoring.rhobs/ 2026-03-20 16:36:41.716035 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/alertmanagers.monitoring.rhobs/metric-storage.yaml 2026-03-20 16:36:41.716050 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/ceilometers.telemetry.openstack.org/ 2026-03-20 16:36:41.716066 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/ceilometers.telemetry.openstack.org/ceilometer.yaml 2026-03-20 16:36:41.716082 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/galeras.mariadb.openstack.org/ 2026-03-20 16:36:41.716103 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/galeras.mariadb.openstack.org/openstack.yaml 2026-03-20 16:36:41.716119 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/keystoneapis.keystone.openstack.org/ 2026-03-20 16:36:41.716135 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/keystoneapis.keystone.openstack.org/keystone.yaml 2026-03-20 16:36:41.716151 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/keystoneservices.keystone.openstack.org/ 2026-03-20 16:36:41.716167 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/keystoneservices.keystone.openstack.org/ceilometer.yaml 2026-03-20 16:36:41.716183 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/mariadbaccounts.mariadb.openstack.org/ 2026-03-20 16:36:41.716198 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/mariadbaccounts.mariadb.openstack.org/keystone.yaml 2026-03-20 16:36:41.716215 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/mariadbaccounts.mariadb.openstack.org/openstack-mariadb-root.yaml 2026-03-20 16:36:41.716231 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/mariadbdatabases.mariadb.openstack.org/ 2026-03-20 16:36:41.716253 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/mariadbdatabases.mariadb.openstack.org/keystone.yaml 2026-03-20 16:36:41.716272 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/memcacheds.memcached.openstack.org/ 2026-03-20 16:36:41.716294 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/memcacheds.memcached.openstack.org/memcached.yaml 2026-03-20 16:36:41.716313 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/metricstorages.telemetry.openstack.org/ 2026-03-20 16:36:41.716382 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/metricstorages.telemetry.openstack.org/metric-storage.yaml 2026-03-20 16:36:41.716403 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/monitoringstacks.monitoring.rhobs/ 2026-03-20 16:36:41.716433 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/monitoringstacks.monitoring.rhobs/metric-storage.yaml 2026-03-20 16:36:41.716450 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/openstackclients.client.openstack.org/ 2026-03-20 16:36:41.716467 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/openstackclients.client.openstack.org/openstackclient.yaml 2026-03-20 16:36:41.716483 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/openstackcontrolplanes.core.openstack.org/ 2026-03-20 16:36:41.716499 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/openstackcontrolplanes.core.openstack.org/openstack.yaml 2026-03-20 16:36:41.716515 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/openstackversions.core.openstack.org/ 2026-03-20 16:36:41.716531 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/openstackversions.core.openstack.org/openstack.yaml 2026-03-20 16:36:41.716548 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/prometheuses.monitoring.rhobs/ 2026-03-20 16:36:41.716564 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/prometheuses.monitoring.rhobs/metric-storage.yaml 2026-03-20 16:36:41.716579 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/prometheusrules.monitoring.rhobs/ 2026-03-20 16:36:41.716596 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/prometheusrules.monitoring.rhobs/metric-storage.yaml 2026-03-20 16:36:41.716612 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/rabbitmqclusters.rabbitmq.com/ 2026-03-20 16:36:41.716632 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/rabbitmqclusters.rabbitmq.com/rabbitmq-notifications.yaml 2026-03-20 16:36:41.716649 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/rabbitmqclusters.rabbitmq.com/rabbitmq.yaml 2026-03-20 16:36:41.716683 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/rabbitmqs.rabbitmq.openstack.org/ 2026-03-20 16:36:41.716706 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/rabbitmqs.rabbitmq.openstack.org/rabbitmq-notifications.yaml 2026-03-20 16:36:41.716723 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/rabbitmqs.rabbitmq.openstack.org/rabbitmq.yaml 2026-03-20 16:36:41.716739 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/ 2026-03-20 16:36:41.716755 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-ceilometer-compute-prom-exporter-tls.yaml 2026-03-20 16:36:41.716771 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-ceilometer-compute-prom-exporter.yaml 2026-03-20 16:36:41.716787 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-ceilometer-ipmi-prom-exporter-tls.yaml 2026-03-20 16:36:41.716803 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-ceilometer-ipmi-prom-exporter.yaml 2026-03-20 16:36:41.716818 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-ceilometer.yaml 2026-03-20 16:36:41.716834 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-kepler.yaml 2026-03-20 16:36:41.716850 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-ksm.yaml 2026-03-20 16:36:41.716869 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-node-exporter-tls.yaml 2026-03-20 16:36:41.716886 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-node-exporter.yaml 2026-03-20 16:36:41.716903 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-openstack-network-exporter-tls.yaml 2026-03-20 16:36:41.716919 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-openstack-network-exporter.yaml 2026-03-20 16:36:41.716935 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-podman-exporter-tls.yaml 2026-03-20 16:36:41.716950 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-podman-exporter.yaml 2026-03-20 16:36:41.716966 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-rabbitmq-notifications.yaml 2026-03-20 16:36:41.716982 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/scrapeconfigs.monitoring.rhobs/telemetry-rabbitmq.yaml 2026-03-20 16:36:41.716998 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/telemetries.telemetry.openstack.org/ 2026-03-20 16:36:41.717015 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/telemetries.telemetry.openstack.org/telemetry.yaml 2026-03-20 16:36:41.717031 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/transporturls.rabbitmq.openstack.org/ 2026-03-20 16:36:41.717047 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/transporturls.rabbitmq.openstack.org/ceilometer-notifications-transport.yaml 2026-03-20 16:36:41.717067 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/crs/transporturls.rabbitmq.openstack.org/keystone-keystone-transport.yaml 2026-03-20 16:36:41.717084 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/daemonset/ 2026-03-20 16:36:41.717100 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/deployments/ 2026-03-20 16:36:41.717115 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/deployments/keystone.yaml 2026-03-20 16:36:41.717132 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/imagestream/ 2026-03-20 16:36:41.717148 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/installplans/ 2026-03-20 16:36:41.717164 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/issuer/ 2026-03-20 16:36:41.717180 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/issuer/rootca-internal.yaml 2026-03-20 16:36:41.717196 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/issuer/rootca-libvirt.yaml 2026-03-20 16:36:41.717211 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/issuer/rootca-ovn.yaml 2026-03-20 16:36:41.717228 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/issuer/rootca-public.yaml 2026-03-20 16:36:41.717243 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/issuer/selfsigned-issuer.yaml 2026-03-20 16:36:41.717259 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/jobs/ 2026-03-20 16:36:41.717274 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/jobs/keystone-cron-29567041.yaml 2026-03-20 16:36:41.717290 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/poddisruptionbudgets/ 2026-03-20 16:36:41.717310 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/poddisruptionbudgets/metric-storage-alertmanager.yaml 2026-03-20 16:36:41.717326 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/ 2026-03-20 16:36:41.717343 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/alertmanager-metric-storage-0/ 2026-03-20 16:36:41.717359 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/alertmanager-metric-storage-0/alertmanager-metric-storage-0-describe 2026-03-20 16:36:41.717375 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/alertmanager-metric-storage-0/logs/ 2026-03-20 16:36:41.717391 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/alertmanager-metric-storage-0/logs/alertmanager.log 2026-03-20 16:36:41.717407 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/alertmanager-metric-storage-0/logs/config-reloader.log 2026-03-20 16:36:41.717424 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/alertmanager-metric-storage-0/logs/init-config-reloader.log 2026-03-20 16:36:41.717440 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/ceilometer-0/ 2026-03-20 16:36:41.717455 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/ceilometer-0/ceilometer-0-describe 2026-03-20 16:36:41.717471 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/ceilometer-0/logs/ 2026-03-20 16:36:41.717487 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/ceilometer-0/logs/ceilometer-central-agent.log 2026-03-20 16:36:41.717503 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/ceilometer-0/logs/ceilometer-notification-agent.log 2026-03-20 16:36:41.717523 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/ceilometer-0/logs/proxy-httpd.log 2026-03-20 16:36:41.717540 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/ceilometer-0/logs/sg-core.log 2026-03-20 16:36:41.717555 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/keystone-5d69f9588b-zsrh7/ 2026-03-20 16:36:41.717571 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/keystone-5d69f9588b-zsrh7/keystone-5d69f9588b-zsrh7-describe 2026-03-20 16:36:41.717587 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/keystone-5d69f9588b-zsrh7/logs/ 2026-03-20 16:36:41.717603 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/keystone-5d69f9588b-zsrh7/logs/keystone-api.log 2026-03-20 16:36:41.717619 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/keystone-cron-29567041-94xtk/ 2026-03-20 16:36:41.717635 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/keystone-cron-29567041-94xtk/keystone-cron-29567041-94xtk-describe 2026-03-20 16:36:41.717651 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/keystone-cron-29567041-94xtk/logs/ 2026-03-20 16:36:41.717683 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/keystone-cron-29567041-94xtk/logs/keystone-cron.log 2026-03-20 16:36:41.717704 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/kube-state-metrics-0/ 2026-03-20 16:36:41.717721 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/kube-state-metrics-0/kube-state-metrics-0-describe 2026-03-20 16:36:41.717737 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/kube-state-metrics-0/logs/ 2026-03-20 16:36:41.717753 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/kube-state-metrics-0/logs/kube-state-metrics.log 2026-03-20 16:36:41.717773 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/memcached-0/ 2026-03-20 16:36:41.717789 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/memcached-0/memcached-0-describe 2026-03-20 16:36:41.717805 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/memcached-0/logs/ 2026-03-20 16:36:41.717820 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/memcached-0/logs/memcached.log.gz 2026-03-20 16:36:41.717836 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/openstack-galera-0/ 2026-03-20 16:36:41.717851 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/openstack-galera-0/openstack-galera-0-describe 2026-03-20 16:36:41.717867 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/openstack-galera-0/logs/ 2026-03-20 16:36:41.717882 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/openstack-galera-0/logs/galera.log 2026-03-20 16:36:41.717898 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/openstack-galera-0/logs/mysql-bootstrap.log 2026-03-20 16:36:41.717914 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/openstackclient/ 2026-03-20 16:36:41.717929 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/openstackclient/openstackclient-describe 2026-03-20 16:36:41.717945 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/openstackclient/logs/ 2026-03-20 16:36:41.717961 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/openstackclient/logs/openstackclient.log 2026-03-20 16:36:41.717976 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/prometheus-metric-storage-0/ 2026-03-20 16:36:41.717999 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/prometheus-metric-storage-0/prometheus-metric-storage-0-describe 2026-03-20 16:36:41.718016 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/prometheus-metric-storage-0/logs/ 2026-03-20 16:36:41.718032 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/prometheus-metric-storage-0/logs/config-reloader.log 2026-03-20 16:36:41.718048 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/prometheus-metric-storage-0/logs/init-config-reloader.log 2026-03-20 16:36:41.718065 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/prometheus-metric-storage-0/logs/prometheus.log 2026-03-20 16:36:41.718081 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/prometheus-metric-storage-0/logs/thanos-sidecar.log 2026-03-20 16:36:41.718097 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/rabbitmq-notifications-server-0/ 2026-03-20 16:36:41.718112 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/rabbitmq-notifications-server-0/rabbitmq-notifications-server-0-describe 2026-03-20 16:36:41.718128 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/rabbitmq-notifications-server-0/logs/ 2026-03-20 16:36:41.718144 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/rabbitmq-notifications-server-0/logs/rabbitmq.log 2026-03-20 16:36:41.718160 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/rabbitmq-notifications-server-0/logs/setup-container.log 2026-03-20 16:36:41.718176 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/rabbitmq-server-0/ 2026-03-20 16:36:41.718192 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/rabbitmq-server-0/rabbitmq-server-0-describe 2026-03-20 16:36:41.718212 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/rabbitmq-server-0/logs/ 2026-03-20 16:36:41.718229 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/rabbitmq-server-0/logs/rabbitmq.log 2026-03-20 16:36:41.718245 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/pods/rabbitmq-server-0/logs/setup-container.log 2026-03-20 16:36:41.718261 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/replicaset/ 2026-03-20 16:36:41.718277 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/replicaset/keystone-5d69f9588b.yaml 2026-03-20 16:36:41.718292 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/replicaset/keystone-8469ccf466.yaml 2026-03-20 16:36:41.718308 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/routes/ 2026-03-20 16:36:41.718324 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/routes/keystone-public.yaml 2026-03-20 16:36:41.718340 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/ 2026-03-20 16:36:41.718356 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/ceilometer/ 2026-03-20 16:36:41.718372 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-compute-config-data.yaml 2026-03-20 16:36:41.718388 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-compute-config-data.yaml-ceilometer.conf 2026-03-20 16:36:41.718404 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-compute-config-data.yaml-custom.conf 2026-03-20 16:36:41.718420 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-compute-scripts.yaml 2026-03-20 16:36:41.718440 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-config-data.yaml 2026-03-20 16:36:41.718457 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-config-data.yaml-ceilometer.conf 2026-03-20 16:36:41.718473 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-config-data.yaml-custom.conf 2026-03-20 16:36:41.718489 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-config-data.yaml-httpd.conf 2026-03-20 16:36:41.718505 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-config-data.yaml-ssl.conf 2026-03-20 16:36:41.718521 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-ipmi-config-data.yaml 2026-03-20 16:36:41.718537 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-ipmi-config-data.yaml-ceilometer.conf 2026-03-20 16:36:41.718553 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-ipmi-config-data.yaml-custom.conf 2026-03-20 16:36:41.718569 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-ipmi-scripts.yaml 2026-03-20 16:36:41.718585 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/ceilometer/ceilometer-scripts.yaml 2026-03-20 16:36:41.718601 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/ceilometer/cert-ceilometer-internal-svc.yaml 2026-03-20 16:36:41.718617 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/ceilometer/rabbitmq-transport-url-ceilometer-notifications-transport.yaml 2026-03-20 16:36:41.718633 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/cinder/ 2026-03-20 16:36:41.718654 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/cinder/cert-cinder-internal-svc.yaml 2026-03-20 16:36:41.718694 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/cinder/cert-cinder-public-route.yaml 2026-03-20 16:36:41.718713 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/cinder/cert-cinder-public-svc.yaml 2026-03-20 16:36:41.718729 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/keystone/ 2026-03-20 16:36:41.718745 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/keystone/cert-keystone-internal-svc.yaml 2026-03-20 16:36:41.718761 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/keystone/cert-keystone-public-route.yaml 2026-03-20 16:36:41.718777 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/keystone/cert-keystone-public-svc.yaml 2026-03-20 16:36:41.718793 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/keystone/keystone-config-data.yaml 2026-03-20 16:36:41.718809 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/keystone/keystone-config-data.yaml-httpd.conf 2026-03-20 16:36:41.718825 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/keystone/keystone-config-data.yaml-keystone.conf 2026-03-20 16:36:41.718841 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/keystone/keystone-config-data.yaml-ssl.conf 2026-03-20 16:36:41.718857 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/keystone/keystone-db-secret.yaml 2026-03-20 16:36:41.718873 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/keystone/keystone-scripts.yaml 2026-03-20 16:36:41.718892 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/keystone/keystone.yaml 2026-03-20 16:36:41.718915 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/keystone/rabbitmq-transport-url-keystone-keystone-transport.yaml 2026-03-20 16:36:41.718931 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/mariadb/ 2026-03-20 16:36:41.718947 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/mariadb/openstack-mariadb-root-db-secret.yaml 2026-03-20 16:36:41.718963 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/metric-storage/ 2026-03-20 16:36:41.718978 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/metric-storage/alertmanager-metric-storage-cluster-tls-config.yaml 2026-03-20 16:36:41.718994 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/metric-storage/alertmanager-metric-storage-generated.yaml 2026-03-20 16:36:41.719010 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/metric-storage/alertmanager-metric-storage-tls-assets-0.yaml 2026-03-20 16:36:41.719027 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/metric-storage/alertmanager-metric-storage-web-config.yaml 2026-03-20 16:36:41.719043 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/metric-storage/cert-metric-storage-prometheus-svc.yaml 2026-03-20 16:36:41.719059 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/metric-storage/metric-storage-prometheus-endpoint.yaml 2026-03-20 16:36:41.719075 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/metric-storage/metric-storage-self-scrape.yaml 2026-03-20 16:36:41.719090 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/metric-storage/prometheus-metric-storage-thanos-prometheus-http-client-file.yaml 2026-03-20 16:36:41.719107 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/metric-storage/prometheus-metric-storage-tls-assets-0.yaml 2026-03-20 16:36:41.719126 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/metric-storage/prometheus-metric-storage-web-config.yaml 2026-03-20 16:36:41.719142 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/metric-storage/prometheus-metric-storage.yaml 2026-03-20 16:36:41.719158 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/ovn/ 2026-03-20 16:36:41.719175 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/ovn/rootca-ovn.yaml 2026-03-20 16:36:41.719191 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/rabbitmq/ 2026-03-20 16:36:41.719207 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/rabbitmq/cert-rabbitmq-notifications-svc.yaml 2026-03-20 16:36:41.719223 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/rabbitmq/cert-rabbitmq-svc.yaml 2026-03-20 16:36:41.719240 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/rabbitmq/rabbitmq-default-user.yaml 2026-03-20 16:36:41.719256 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/rabbitmq/rabbitmq-default-user.yaml-default_user.conf 2026-03-20 16:36:41.719271 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/rabbitmq/rabbitmq-erlang-cookie.yaml 2026-03-20 16:36:41.719288 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/rabbitmq/rabbitmq-notifications-default-user.yaml 2026-03-20 16:36:41.719304 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/rabbitmq/rabbitmq-notifications-default-user.yaml-default_user.conf 2026-03-20 16:36:41.719320 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/rabbitmq/rabbitmq-notifications-erlang-cookie.yaml 2026-03-20 16:36:41.719340 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/rabbitmq/rabbitmq-transport-url-ceilometer-notifications-transport.yaml 2026-03-20 16:36:41.719357 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/rabbitmq/rabbitmq-transport-url-keystone-keystone-transport.yaml 2026-03-20 16:36:41.719373 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/watcher/ 2026-03-20 16:36:41.719389 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/watcher/ac-watcher-secret.yaml 2026-03-20 16:36:41.719405 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/watcher/cert-watcher-internal-svc.yaml 2026-03-20 16:36:41.719421 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/secrets/watcher/cert-watcher-public-svc.yaml 2026-03-20 16:36:41.719438 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/services/ 2026-03-20 16:36:41.719454 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/services/alertmanager-operated.yaml 2026-03-20 16:36:41.719471 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/services/ceilometer-internal.yaml 2026-03-20 16:36:41.719487 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/services/keystone-internal.yaml 2026-03-20 16:36:41.719503 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/services/keystone-public.yaml 2026-03-20 16:36:41.719518 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/services/kube-state-metrics.yaml 2026-03-20 16:36:41.719534 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/services/memcached.yaml 2026-03-20 16:36:41.719549 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/services/metric-storage-alertmanager.yaml 2026-03-20 16:36:41.719569 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/services/metric-storage-prometheus.yaml 2026-03-20 16:36:41.719586 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/services/metric-storage-thanos-sidecar.yaml 2026-03-20 16:36:41.719601 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/services/openstack-galera.yaml 2026-03-20 16:36:41.719617 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/services/openstack.yaml 2026-03-20 16:36:41.719633 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/services/prometheus-operated.yaml 2026-03-20 16:36:41.719649 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/services/rabbitmq-nodes.yaml 2026-03-20 16:36:41.719691 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/services/rabbitmq-notifications-nodes.yaml 2026-03-20 16:36:41.719714 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/services/rabbitmq-notifications.yaml 2026-03-20 16:36:41.719731 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/services/rabbitmq.yaml 2026-03-20 16:36:41.719747 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/statefulset/ 2026-03-20 16:36:41.719763 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/statefulset/alertmanager-metric-storage.yaml 2026-03-20 16:36:41.719779 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/statefulset/ceilometer.yaml 2026-03-20 16:36:41.719795 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/statefulset/kube-state-metrics.yaml 2026-03-20 16:36:41.719811 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/statefulset/memcached.yaml 2026-03-20 16:36:41.719830 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/statefulset/openstack-galera.yaml 2026-03-20 16:36:41.719847 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/statefulset/prometheus-metric-storage.yaml 2026-03-20 16:36:41.719863 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/statefulset/rabbitmq-notifications-server.yaml 2026-03-20 16:36:41.719879 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/statefulset/rabbitmq-server.yaml 2026-03-20 16:36:41.719895 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/namespaces/watcher-kuttl-default/subscriptions/ 2026-03-20 16:36:41.719911 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/network/ 2026-03-20 16:36:41.719927 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/network/l2advertisement.yaml 2026-03-20 16:36:41.719943 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/network/ipaddresspools/ 2026-03-20 16:36:41.719959 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/network/ipaddresspools/ctlplane.yaml 2026-03-20 16:36:41.719975 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/network/ipaddresspools/designateext.yaml 2026-03-20 16:36:41.719991 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/network/ipaddresspools/internalapi.yaml 2026-03-20 16:36:41.720007 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/network/ipaddresspools/storage.yaml 2026-03-20 16:36:41.720023 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/network/ipaddresspools/tenant.yaml 2026-03-20 16:36:41.720038 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/network/nnce/ 2026-03-20 16:36:41.720053 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/network/nnce/crc.ens7-crc.yaml 2026-03-20 16:36:41.720073 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/network/nncp/ 2026-03-20 16:36:41.720090 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/network/nncp/ens7-crc.yaml 2026-03-20 16:36:41.720106 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/network/nns/ 2026-03-20 16:36:41.720123 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/network/nns/crc.yaml 2026-03-20 16:36:41.720139 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/nodes/ 2026-03-20 16:36:41.720154 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/nodes/crc.describe 2026-03-20 16:36:41.720170 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/nodes/crc.yaml 2026-03-20 16:36:41.720185 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/sos-reports/ 2026-03-20 16:36:41.720201 | controller | cL+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/sos-reports/sos-report- -> ../_all_nodes/sosreport- 2026-03-20 16:36:41.720217 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/sos-reports/_all_nodes/ 2026-03-20 16:36:41.720233 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/webhooks/ 2026-03-20 16:36:41.720249 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/webhooks/mutating/ 2026-03-20 16:36:41.720264 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/webhooks/mutating/cert-manager-webhook.yaml 2026-03-20 16:36:41.720280 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/webhooks/mutating/infra-operator-mutating-webhook-configuration.yaml 2026-03-20 16:36:41.720296 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/webhooks/mutating/mwatcher-v1beta1.kb.io-xbgj2.yaml 2026-03-20 16:36:41.720312 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/webhooks/mutating/mwatcherapi-v1beta1.kb.io-dvgb4.yaml 2026-03-20 16:36:41.720333 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/webhooks/mutating/mwatcherapplier-v1beta1.kb.io-88hfv.yaml 2026-03-20 16:36:41.720350 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/webhooks/mutating/mwatcherdecisionengine-v1beta1.kb.io-qltbk.yaml 2026-03-20 16:36:41.720366 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/webhooks/mutating/nmstate.yaml 2026-03-20 16:36:41.720382 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/webhooks/mutating/openstack-baremetal-operator-mutating-webhook-configuration.yaml 2026-03-20 16:36:41.720398 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/webhooks/mutating/openstack-operator-mutating-webhook-configuration.yaml 2026-03-20 16:36:41.720415 | controller | cd+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/webhooks/validating/ 2026-03-20 16:36:41.720431 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/webhooks/validating/alertmanagerconfigs.monitoring.rhobs-bcbjn.yaml 2026-03-20 16:36:41.720464 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/webhooks/validating/bfdprofilevalidationwebhook.metallb.io-k74lk.yaml 2026-03-20 16:36:41.720480 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/webhooks/validating/bgpadvertisementvalidationwebhook.metallb.io-wwtn4.yaml 2026-03-20 16:36:41.720497 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/webhooks/validating/bgppeersvalidationwebhook.metallb.io-tkbxd.yaml 2026-03-20 16:36:41.720513 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/webhooks/validating/cert-manager-webhook.yaml 2026-03-20 16:36:41.720529 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/webhooks/validating/communityvalidationwebhook.metallb.io-gtnlg.yaml 2026-03-20 16:36:41.720546 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/webhooks/validating/controlplanemachineset.machine.openshift.io.yaml 2026-03-20 16:36:41.720562 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/webhooks/validating/frr-k8s-validating-webhook-configuration.yaml 2026-03-20 16:36:41.720582 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/webhooks/validating/infra-operator-validating-webhook-configuration.yaml 2026-03-20 16:36:41.720598 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/webhooks/validating/ipaddresspoolvalidationwebhook.metallb.io-6bszv.yaml 2026-03-20 16:36:41.720615 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/webhooks/validating/l2advertisementvalidationwebhook.metallb.io-9zcs4.yaml 2026-03-20 16:36:41.720631 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/webhooks/validating/metallbvalidationwebhook.metallb.io-d6dz6.yaml 2026-03-20 16:36:41.720648 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/webhooks/validating/monitoringconfigmaps.openshift.io.yaml 2026-03-20 16:36:41.720664 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/webhooks/validating/multus.openshift.io.yaml 2026-03-20 16:36:41.720700 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/webhooks/validating/network-node-identity.openshift.io.yaml 2026-03-20 16:36:41.720717 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/webhooks/validating/openstack-baremetal-operator-validating-webhook-configuration.yaml 2026-03-20 16:36:41.720733 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/webhooks/validating/openstack-operator-validating-webhook-configuration.yaml 2026-03-20 16:36:41.720750 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/webhooks/validating/prometheusrules.monitoring.rhobs-phn89.yaml 2026-03-20 16:36:41.720767 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/webhooks/validating/vwatcher-v1beta1.kb.io-ptv7n.yaml 2026-03-20 16:36:41.720783 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/webhooks/validating/vwatcherapi-v1beta1.kb.io-rpnvq.yaml 2026-03-20 16:36:41.720799 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/webhooks/validating/vwatcherapplier-v1beta1.kb.io-9kpk7.yaml 2026-03-20 16:36:41.720816 | controller | >f+++++++++ ci-framework-data/logs/openstack-must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-58cb6caac54870493f224cc37a6252a09ea9ccbbda2851efaaba72cc8a6f1339/webhooks/validating/vwatcherdecisionengine-v1beta1.kb.io-crpz6.yaml 2026-03-20 16:36:41.720832 | controller | cd+++++++++ registries.conf.d/ 2026-03-20 16:36:41.720853 | controller | >f+++++++++ registries.conf.d/000-shortnames.conf 2026-03-20 16:36:41.720871 | controller | cd+++++++++ system-config/ 2026-03-20 16:36:41.720888 | controller | cd+++++++++ system-config/libvirt/ 2026-03-20 16:36:41.720904 | controller | >f+++++++++ system-config/libvirt/libvirt-admin.conf 2026-03-20 16:36:41.720921 | controller | >f+++++++++ system-config/libvirt/libvirt.conf 2026-03-20 16:36:42.171660 | controller | ok 2026-03-20 16:36:42.703832 | controller | ok 2026-03-20 16:36:42.835777 | 2026-03-20 16:36:42.835888 | TASK [Return artifact to Zuul] 2026-03-20 16:36:42.861220 | controller | skipping: Conditional result was False 2026-03-20 16:36:42.955090 | 2026-03-20 16:36:42.955183 | PLAY [Run ci/playbooks/collect-logs.yml on CRC host] 2026-03-20 16:36:42.974908 | 2026-03-20 16:36:42.974970 | TASK [Get kubelet journalctl logs] 2026-03-20 16:36:44.058370 | crc | Output suppressed because no_log was given 2026-03-20 16:36:44.073315 | 2026-03-20 16:36:44.073446 | TASK [Compress logs bigger than 2MB] 2026-03-20 16:36:44.420192 | crc | gzip: /var/home/core/zuul-output/logs/kubelet.log.gz already exists; not overwritten 2026-03-20 16:36:44.637166 | crc | ERROR 2026-03-20 16:36:44.637387 | crc | { 2026-03-20 16:36:44.637447 | crc | "delta": "0:00:00.009285", 2026-03-20 16:36:44.637488 | crc | "end": "2026-03-20 16:36:44.421092", 2026-03-20 16:36:44.637524 | crc | "msg": "non-zero return code", 2026-03-20 16:36:44.637560 | crc | "rc": 1, 2026-03-20 16:36:44.637595 | crc | "start": "2026-03-20 16:36:44.411807" 2026-03-20 16:36:44.637629 | crc | } 2026-03-20 16:36:44.637719 | crc | ERROR: Ignoring Errors 2026-03-20 16:36:44.653100 | 2026-03-20 16:36:44.653231 | TASK [Copy files from workspace on node] 2026-03-20 16:36:44.686473 | crc | ok 2026-03-20 16:36:44.721693 | 2026-03-20 16:36:44.721791 | TASK [fetch-output : Set log path for multiple nodes] 2026-03-20 16:36:44.766185 | crc | ok 2026-03-20 16:36:44.775525 | 2026-03-20 16:36:44.775614 | TASK [fetch-output : Set log path for single node] 2026-03-20 16:36:44.800100 | crc | skipping: Conditional result was False 2026-03-20 16:36:44.812053 | 2026-03-20 16:36:44.812159 | LOOP [fetch-output : Ensure local output dirs] 2026-03-20 16:36:45.098931 | crc -> localhost | ok: "/var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/work/logs/crc" 2026-03-20 16:36:45.099338 | crc -> localhost | changed: All items complete 2026-03-20 16:36:45.099395 | 2026-03-20 16:36:45.307495 | crc -> localhost | ok: "/var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/work/artifacts" 2026-03-20 16:36:45.519138 | crc -> localhost | ok: "/var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/work/docs" 2026-03-20 16:36:45.538486 | 2026-03-20 16:36:45.538635 | LOOP [fetch-output : Collect logs, artifacts and docs] 2026-03-20 16:36:46.312624 | crc | changed: 2026-03-20 16:36:46.313206 | crc | .d..t...... ./ 2026-03-20 16:36:46.313278 | crc | >f+++++++++ kubelet.log 2026-03-20 16:36:46.874834 | crc | ok 2026-03-20 16:36:47.435351 | crc | ok 2026-03-20 16:36:47.510893 | 2026-03-20 16:36:47.511056 | PLAY RECAP 2026-03-20 16:36:47.511125 | controller | ok: 15 changed: 11 unreachable: 0 failed: 0 skipped: 7 rescued: 0 ignored: 1 2026-03-20 16:36:47.511169 | crc | ok: 5 changed: 3 unreachable: 0 failed: 0 skipped: 1 rescued: 0 ignored: 1 2026-03-20 16:36:47.511200 | 2026-03-20 16:36:47.654214 | POST-RUN END RESULT_NORMAL: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/collect-logs.yml@main] 2026-03-20 16:36:47.663534 | POST-RUN START: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/multinode-autohold.yml@main] 2026-03-20 16:36:48.276039 | 2026-03-20 16:36:48.276155 | PLAY [Run ci/playbooks/multinode-autohold.yml] 2026-03-20 16:36:48.353805 | 2026-03-20 16:36:48.353932 | TASK [Verify if "success" flag exists after successful tests execution] 2026-03-20 16:36:49.203208 | controller | ok 2026-03-20 16:36:49.228058 | 2026-03-20 16:36:49.228129 | TASK [Fetch existing autoholds from zuul] 2026-03-20 16:36:49.329588 | controller | ok 2026-03-20 16:36:49.386293 | 2026-03-20 16:36:49.386363 | TASK [krb_request : Perform the request] 2026-03-20 16:36:50.149623 | controller | ok 2026-03-20 16:36:50.236630 | 2026-03-20 16:36:50.236769 | TASK [Check if any autohold matches] 2026-03-20 16:36:50.342710 | controller | skipping: Conditional result was False 2026-03-20 16:36:50.471365 | 2026-03-20 16:36:50.471464 | PLAY RECAP 2026-03-20 16:36:50.471517 | controller | ok: 2 changed: 0 unreachable: 0 failed: 0 skipped: 1 rescued: 0 ignored: 0 2026-03-20 16:36:50.471544 | 2026-03-20 16:36:50.582778 | POST-RUN END RESULT_NORMAL: [untrusted : github.com/openstack-k8s-operators/ci-framework/ci/playbooks/multinode-autohold.yml@main] 2026-03-20 16:36:50.587183 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/crc/print-machineconfigpool.yaml@master] 2026-03-20 16:36:51.144899 | 2026-03-20 16:36:51.145019 | PLAY [crc] 2026-03-20 16:36:51.175534 | 2026-03-20 16:36:51.175731 | TASK [Print machineconfigpool] 2026-03-20 16:36:51.987103 | crc | NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE 2026-03-20 16:36:51.992468 | crc | master rendered-master-f83a149c7490c3b0bf850df839837383 False False True 1 0 0 1 390d 2026-03-20 16:36:51.992511 | crc | worker rendered-worker-e1a74fb57786f7a3a72e33bc3650d695 True False False 0 0 0 0 390d 2026-03-20 16:36:52.215503 | crc | changed 2026-03-20 16:36:52.268470 | 2026-03-20 16:36:52.268541 | PLAY RECAP 2026-03-20 16:36:52.268601 | crc | ok: 1 changed: 1 unreachable: 0 failed: 0 skipped: 0 rescued: 0 ignored: 0 2026-03-20 16:36:52.268629 | 2026-03-20 16:36:52.362655 | POST-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/crc/print-machineconfigpool.yaml@master] 2026-03-20 16:36:52.372315 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/crc/compress-crc-cloud-logs.yaml@master] 2026-03-20 16:36:52.926464 | 2026-03-20 16:36:52.926567 | PLAY [crc] 2026-03-20 16:36:52.952589 | 2026-03-20 16:36:52.952686 | TASK [Create compressed log tar gz file - crc-cloud logs] 2026-03-20 16:36:53.668799 | crc | tar: Removing leading `/' from member names 2026-03-20 16:36:53.668849 | crc | tar: /var/home/core/workdir/*log: Cannot stat: No such file or directory 2026-03-20 16:36:53.670857 | crc | tar: Exiting with failure status due to previous errors 2026-03-20 16:36:53.994661 | crc | ok 2026-03-20 16:36:54.009531 | 2026-03-20 16:36:54.009692 | TASK [Copy crc-cloud logs] 2026-03-20 16:36:54.521738 | crc | cp: cannot stat '/var/home/core/workdir/*log': No such file or directory 2026-03-20 16:36:54.575539 | crc | ok 2026-03-20 16:36:54.625606 | 2026-03-20 16:36:54.625786 | PLAY RECAP 2026-03-20 16:36:54.625853 | crc | ok: 2 changed: 0 unreachable: 0 failed: 0 skipped: 0 rescued: 0 ignored: 0 2026-03-20 16:36:54.625880 | 2026-03-20 16:36:54.720777 | POST-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/crc/compress-crc-cloud-logs.yaml@master] 2026-03-20 16:36:54.730338 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/crc/compress-logs.yaml@master] 2026-03-20 16:36:55.312687 | 2026-03-20 16:36:55.312800 | PLAY [all] 2026-03-20 16:36:55.401658 | 2026-03-20 16:36:55.401789 | TASK [Create compressed log tar gz file - ci-framework-data] 2026-03-20 16:36:56.224173 | crc | tar: Removing leading `/' from member names 2026-03-20 16:36:56.224216 | crc | tar: /var/home/core/ci-framework-data: Cannot stat: No such file or directory 2026-03-20 16:36:56.226848 | crc | tar: Exiting with failure status due to previous errors 2026-03-20 16:36:56.229077 | crc | mv: cannot move '/var/home/core/ci-framework-data-crc-all-logs.tar.gz' to '/var/home/core/ci-framework-data/logs/ci-framework-data-crc-all-logs.tar.gz': No such file or directory 2026-03-20 16:36:56.231206 | crc | chown: cannot access '/var/home/core/ci-framework-data/logs/ci-framework-data-crc-all-logs.tar.gz': No such file or directory 2026-03-20 16:36:56.074798 | controller | tar: Removing leading `/' from member names 2026-03-20 16:37:05.080360 | crc | ERROR 2026-03-20 16:37:05.080642 | crc | { 2026-03-20 16:37:05.080796 | crc | "delta": "0:00:00.016997", 2026-03-20 16:37:05.080847 | crc | "end": "2026-03-20 16:36:56.231713", 2026-03-20 16:37:05.080885 | crc | "msg": "non-zero return code", 2026-03-20 16:37:05.080921 | crc | "rc": 1, 2026-03-20 16:37:05.080958 | crc | "start": "2026-03-20 16:36:56.214716" 2026-03-20 16:37:05.080993 | crc | } 2026-03-20 16:37:05.081049 | crc | ERROR: Ignoring Errors 2026-03-20 16:37:05.156356 | controller | ok 2026-03-20 16:37:05.182592 | 2026-03-20 16:37:05.182652 | TASK [Create compressed log tar gz file - zuul-output] 2026-03-20 16:37:05.626509 | controller | tar: Removing leading `/' from member names 2026-03-20 16:37:05.734292 | crc | tar: Removing leading `/' from member names 2026-03-20 16:37:08.363405 | crc | ok 2026-03-20 16:37:08.405976 | controller | ok 2026-03-20 16:37:08.532336 | 2026-03-20 16:37:08.532441 | PLAY RECAP 2026-03-20 16:37:08.532489 | controller | ok: 2 changed: 0 unreachable: 0 failed: 0 skipped: 0 rescued: 0 ignored: 0 2026-03-20 16:37:08.532514 | crc | ok: 2 changed: 0 unreachable: 0 failed: 0 skipped: 0 rescued: 0 ignored: 1 2026-03-20 16:37:08.532532 | 2026-03-20 16:37:08.630875 | POST-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/crc/compress-logs.yaml@master] 2026-03-20 16:37:08.637915 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/crc/bootstraped-ci-network-cleanup.yml@master] 2026-03-20 16:37:09.214876 | 2026-03-20 16:37:09.214984 | PLAY [Run playbooks/crc/bootstraped-ci-network-cleanup.yml] 2026-03-20 16:37:09.290460 | 2026-03-20 16:37:09.290564 | TASK [Create openstack config dir] 2026-03-20 16:37:09.864677 | controller | ok 2026-03-20 16:37:09.887546 | 2026-03-20 16:37:09.887618 | TASK [Check for cleanup skip flag file presence] 2026-03-20 16:37:10.290592 | controller | ok 2026-03-20 16:37:10.314248 | 2026-03-20 16:37:10.314312 | TASK [Generate clouds config from cloud_secrets secret] 2026-03-20 16:37:11.242816 | controller | changed 2026-03-20 16:37:11.290269 | 2026-03-20 16:37:11.290408 | LOOP [Delete ports from each host] 2026-03-20 16:37:11.454636 | controller | ok: "crc" 2026-03-20 16:37:14.419032 | controller | ok: "controller" 2026-03-20 16:37:14.479896 | controller | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/trusted/project_0/review.rdoproject.org/config/playbooks/crc/bootstraped-ci-network-cleanup-instance-ports.yml 2026-03-20 16:37:14.484449 | controller | included: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/trusted/project_0/review.rdoproject.org/config/playbooks/crc/bootstraped-ci-network-cleanup-instance-ports.yml 2026-03-20 16:37:14.513348 | 2026-03-20 16:37:14.513408 | TASK [Fetch port info] 2026-03-20 16:37:17.030082 | controller | admin_state_up: true 2026-03-20 16:37:17.030519 | controller | allowed_address_pairs: [] 2026-03-20 16:37:17.030928 | controller | binding_host_id: null 2026-03-20 16:37:17.031159 | controller | binding_profile: {} 2026-03-20 16:37:17.031385 | controller | binding_vif_details: {} 2026-03-20 16:37:17.031604 | controller | binding_vif_type: null 2026-03-20 16:37:17.031907 | controller | binding_vnic_type: normal 2026-03-20 16:37:17.032156 | controller | created_at: '2026-03-20T15:35:59Z' 2026-03-20 16:37:17.032455 | controller | data_plane_status: null 2026-03-20 16:37:17.032657 | controller | description: '' 2026-03-20 16:37:17.032936 | controller | device_id: a34aaf21-bc48-4d4d-b0c3-48ef72d66bb9 2026-03-20 16:37:17.033153 | controller | device_owner: compute:nova 2026-03-20 16:37:17.033416 | controller | device_profile: null 2026-03-20 16:37:17.034086 | controller | dns_assignment: 2026-03-20 16:37:17.034313 | controller | - fqdn: host-192-168-122-10.openstacklocal. 2026-03-20 16:37:17.034325 | controller | hostname: host-192-168-122-10 2026-03-20 16:37:17.034333 | controller | ip_address: 192.168.122.10 2026-03-20 16:37:17.034340 | controller | dns_domain: '' 2026-03-20 16:37:17.034538 | controller | dns_name: '' 2026-03-20 16:37:17.034862 | controller | extra_dhcp_opts: [] 2026-03-20 16:37:17.035357 | controller | fixed_ips: 2026-03-20 16:37:17.035579 | controller | - ip_address: 192.168.122.10 2026-03-20 16:37:17.035591 | controller | subnet_id: fbbf2376-14dd-4853-abbb-f430e1538dd0 2026-03-20 16:37:17.035600 | controller | hardware_offload_type: null 2026-03-20 16:37:17.035790 | controller | hints: '' 2026-03-20 16:37:17.036023 | controller | id: 971ec772-62a8-4bf1-9450-4da0866ed20e 2026-03-20 16:37:17.036250 | controller | ip_allocation: immediate 2026-03-20 16:37:17.036493 | controller | mac_address: fa:16:3e:02:5c:b8 2026-03-20 16:37:17.036904 | controller | name: crc-a34aaf21-bc48-4d4d-b0c3-48ef72d66bb9 2026-03-20 16:37:17.037279 | controller | network_id: adbac40f-789f-4953-a393-3ccea9f4523a 2026-03-20 16:37:17.037518 | controller | numa_affinity_policy: null 2026-03-20 16:37:17.037865 | controller | port_security_enabled: false 2026-03-20 16:37:17.038124 | controller | project_id: 4b633c451ac74233be3721a3635275e5 2026-03-20 16:37:17.038410 | controller | propagate_uplink_status: null 2026-03-20 16:37:17.038639 | controller | resource_request: null 2026-03-20 16:37:17.039009 | controller | revision_number: 4 2026-03-20 16:37:17.039270 | controller | qos_network_policy_id: null 2026-03-20 16:37:17.039475 | controller | qos_policy_id: null 2026-03-20 16:37:17.039748 | controller | security_group_ids: [] 2026-03-20 16:37:17.039971 | controller | status: ACTIVE 2026-03-20 16:37:17.040182 | controller | tags: [] 2026-03-20 16:37:17.040386 | controller | trunk_details: null 2026-03-20 16:37:17.040598 | controller | trusted: null 2026-03-20 16:37:17.040863 | controller | updated_at: '2026-03-20T15:36:11Z' 2026-03-20 16:37:17.223266 | controller | changed 2026-03-20 16:37:17.246823 | 2026-03-20 16:37:17.246888 | TASK [Set port info yaml for further usage] 2026-03-20 16:37:17.322652 | controller | ok 2026-03-20 16:37:17.373757 | 2026-03-20 16:37:17.373898 | TASK [Detach port] 2026-03-20 16:37:20.496054 | controller | changed 2026-03-20 16:37:20.564247 | 2026-03-20 16:37:20.564389 | LOOP [Delete port association with trunk network] 2026-03-20 16:37:20.635062 | 2026-03-20 16:37:20.635238 | LOOP [Delete trunk sub ports (tagged ones)] 2026-03-20 16:37:20.694858 | 2026-03-20 16:37:20.695027 | TASK [Delete the trunk network itself] 2026-03-20 16:37:21.354633 | controller | skipping: Conditional result was False 2026-03-20 16:37:21.407531 | 2026-03-20 16:37:21.407633 | TASK [Remove the parent/trunk port itself] 2026-03-20 16:37:36.086223 | controller | changed 2026-03-20 16:37:36.119749 | 2026-03-20 16:37:36.119864 | TASK [Fetch port info] 2026-03-20 16:37:44.100895 | controller | admin_state_up: true 2026-03-20 16:37:44.101060 | controller | allowed_address_pairs: [] 2026-03-20 16:37:44.101219 | controller | binding_host_id: null 2026-03-20 16:37:44.101348 | controller | binding_profile: {} 2026-03-20 16:37:44.101501 | controller | binding_vif_details: {} 2026-03-20 16:37:44.101659 | controller | binding_vif_type: null 2026-03-20 16:37:44.101875 | controller | binding_vnic_type: normal 2026-03-20 16:37:44.102029 | controller | created_at: '2026-03-20T15:35:44Z' 2026-03-20 16:37:44.102160 | controller | data_plane_status: null 2026-03-20 16:37:44.102292 | controller | description: '' 2026-03-20 16:37:44.102423 | controller | device_id: ca1c710b-d9a5-466b-a8d0-123e17aad387 2026-03-20 16:37:44.102533 | controller | device_owner: compute:nova 2026-03-20 16:37:44.102629 | controller | device_profile: null 2026-03-20 16:37:44.103042 | controller | dns_assignment: 2026-03-20 16:37:44.103136 | controller | - fqdn: host-192-168-122-11.openstacklocal. 2026-03-20 16:37:44.103144 | controller | hostname: host-192-168-122-11 2026-03-20 16:37:44.103151 | controller | ip_address: 192.168.122.11 2026-03-20 16:37:44.103160 | controller | dns_domain: '' 2026-03-20 16:37:44.103233 | controller | dns_name: '' 2026-03-20 16:37:44.103347 | controller | extra_dhcp_opts: [] 2026-03-20 16:37:44.103595 | controller | fixed_ips: 2026-03-20 16:37:44.103711 | controller | - ip_address: 192.168.122.11 2026-03-20 16:37:44.103722 | controller | subnet_id: fbbf2376-14dd-4853-abbb-f430e1538dd0 2026-03-20 16:37:44.103730 | controller | hardware_offload_type: null 2026-03-20 16:37:44.103804 | controller | hints: '' 2026-03-20 16:37:44.103919 | controller | id: 30e4281f-5f88-42bc-a0da-d9e23bfff189 2026-03-20 16:37:44.104022 | controller | ip_allocation: immediate 2026-03-20 16:37:44.104125 | controller | mac_address: fa:16:3e:0d:21:04 2026-03-20 16:37:44.104244 | controller | name: controller-ca1c710b-d9a5-466b-a8d0-123e17aad387 2026-03-20 16:37:44.104361 | controller | network_id: adbac40f-789f-4953-a393-3ccea9f4523a 2026-03-20 16:37:44.104464 | controller | numa_affinity_policy: null 2026-03-20 16:37:44.104606 | controller | port_security_enabled: false 2026-03-20 16:37:44.104742 | controller | project_id: 4b633c451ac74233be3721a3635275e5 2026-03-20 16:37:44.104846 | controller | propagate_uplink_status: null 2026-03-20 16:37:44.104944 | controller | resource_request: null 2026-03-20 16:37:44.105099 | controller | revision_number: 4 2026-03-20 16:37:44.105201 | controller | qos_network_policy_id: null 2026-03-20 16:37:44.105298 | controller | qos_policy_id: null 2026-03-20 16:37:44.105398 | controller | security_group_ids: [] 2026-03-20 16:37:44.105489 | controller | status: ACTIVE 2026-03-20 16:37:44.105579 | controller | tags: [] 2026-03-20 16:37:44.105674 | controller | trunk_details: null 2026-03-20 16:37:44.105783 | controller | trusted: null 2026-03-20 16:37:44.105900 | controller | updated_at: '2026-03-20T15:35:58Z' 2026-03-20 16:37:44.321036 | controller | changed 2026-03-20 16:37:44.352278 | 2026-03-20 16:37:44.352345 | TASK [Set port info yaml for further usage] 2026-03-20 16:37:44.544086 | controller | ok 2026-03-20 16:37:44.567123 | 2026-03-20 16:37:44.567185 | TASK [Detach port] 2026-03-20 16:37:56.730205 | controller | changed 2026-03-20 16:37:56.785065 | 2026-03-20 16:37:56.785286 | LOOP [Delete port association with trunk network] 2026-03-20 16:37:56.906411 | 2026-03-20 16:37:56.906635 | LOOP [Delete trunk sub ports (tagged ones)] 2026-03-20 16:37:56.984428 | 2026-03-20 16:37:56.984680 | TASK [Delete the trunk network itself] 2026-03-20 16:37:57.689443 | controller | skipping: Conditional result was False 2026-03-20 16:37:57.744017 | 2026-03-20 16:37:57.744182 | TASK [Remove the parent/trunk port itself] 2026-03-20 16:38:11.418705 | controller | changed 2026-03-20 16:38:11.450727 | 2026-03-20 16:38:11.450818 | TASK [Dettach router from subnet] 2026-03-20 16:38:12.076651 | controller | skipping: Conditional result was False 2026-03-20 16:38:12.158724 | 2026-03-20 16:38:12.158879 | TASK [Delete router] 2026-03-20 16:38:12.779177 | controller | skipping: Conditional result was False 2026-03-20 16:38:12.868482 | 2026-03-20 16:38:12.868619 | TASK [Delete subnet] 2026-03-20 16:38:26.535049 | controller | changed 2026-03-20 16:38:26.575367 | 2026-03-20 16:38:26.575528 | TASK [Delete network] 2026-03-20 16:38:40.318750 | controller | changed 2026-03-20 16:38:40.350427 | 2026-03-20 16:38:40.350517 | TASK [Remove clouds.yml secret] 2026-03-20 16:38:40.630607 | controller | changed 2026-03-20 16:38:40.763082 | 2026-03-20 16:38:40.763195 | PLAY RECAP 2026-03-20 16:38:40.763239 | controller | ok: 16 changed: 10 unreachable: 0 failed: 0 skipped: 8 rescued: 0 ignored: 0 2026-03-20 16:38:40.763259 | 2026-03-20 16:38:40.870066 | POST-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/crc/bootstraped-ci-network-cleanup.yml@master] 2026-03-20 16:38:40.878587 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/ci-framework-rdo-base/dlrn/dlrn-report.yaml@master] 2026-03-20 16:38:41.484127 | 2026-03-20 16:38:41.484267 | PLAY [Report DLRN results after job run] 2026-03-20 16:38:41.567858 | 2026-03-20 16:38:41.568001 | TASK [Report job status to DLRN] 2026-03-20 16:38:41.633936 | controller | ok 2026-03-20 16:38:41.698363 | 2026-03-20 16:38:41.698509 | TASK [dlrn_report : Warning if DLRN is not running] 2026-03-20 16:38:41.793782 | controller | skipping: Conditional result was False 2026-03-20 16:38:41.820550 | 2026-03-20 16:38:41.820696 | TASK [dlrn_report : Install dlrnapi-client shyaml package] 2026-03-20 16:38:41.864904 | controller | skipping: Conditional result was False 2026-03-20 16:38:41.894445 | 2026-03-20 16:38:41.894588 | TASK [dlrn_report : Install kinit related package] 2026-03-20 16:38:41.952003 | controller | skipping: Conditional result was False 2026-03-20 16:38:41.994847 | 2026-03-20 16:38:41.994982 | TASK [dlrn_report : Install dlrn kerberos related packages] 2026-03-20 16:38:42.039853 | controller | skipping: Conditional result was False 2026-03-20 16:38:42.119472 | 2026-03-20 16:38:42.119609 | TASK [Set zuul-log-path fact] 2026-03-20 16:38:42.163878 | controller | skipping: Conditional result was False 2026-03-20 16:38:42.188778 | 2026-03-20 16:38:42.188854 | TASK [dlrn_report : Set the the value of cifmw_repo_setup_promotion when multiple dlrn tags are used] 2026-03-20 16:38:42.233267 | controller | skipping: Conditional result was False 2026-03-20 16:38:42.268171 | 2026-03-20 16:38:42.268301 | TASK [Get hash related data from repo_setup role] 2026-03-20 16:38:42.313602 | controller | skipping: Conditional result was False 2026-03-20 16:38:42.349303 | 2026-03-20 16:38:42.349417 | TASK [dlrn_report : Perform kinit for DLRN kerberos authentication] 2026-03-20 16:38:43.016229 | controller | skipping: Conditional result was False 2026-03-20 16:38:43.068448 | 2026-03-20 16:38:43.068627 | TASK [dlrn_report : Set empty value for dlrnapi password] 2026-03-20 16:38:43.124333 | controller | skipping: Conditional result was False 2026-03-20 16:38:43.159776 | 2026-03-20 16:38:43.159944 | TASK [dlrn_report : Report results to dlrn for the tested hash] 2026-03-20 16:38:43.810436 | controller | skipping: Conditional result was False 2026-03-20 16:38:44.015292 | 2026-03-20 16:38:44.015403 | PLAY RECAP 2026-03-20 16:38:44.015440 | controller | ok: 0 changed: 0 unreachable: 0 failed: 0 skipped: 10 rescued: 0 ignored: 0 2026-03-20 16:38:44.015460 | 2026-03-20 16:38:44.110201 | POST-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/ci-framework-rdo-base/dlrn/dlrn-report.yaml@master] 2026-03-20 16:38:44.120180 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/base-minimal/post-ssh.yaml@master] 2026-03-20 16:38:44.728553 | 2026-03-20 16:38:44.728688 | PLAY [all] 2026-03-20 16:38:44.839302 | 2026-03-20 16:38:44.839451 | TASK [include_role : fetch-output] 2026-03-20 16:38:44.911265 | controller | ok 2026-03-20 16:38:44.918043 | crc | ok 2026-03-20 16:38:44.978249 | 2026-03-20 16:38:44.978402 | TASK [fetch-output : Set log path for multiple nodes] 2026-03-20 16:38:45.050564 | controller | ok 2026-03-20 16:38:45.057425 | crc | ok 2026-03-20 16:38:45.079278 | 2026-03-20 16:38:45.079343 | TASK [fetch-output : Set log path for single node] 2026-03-20 16:38:45.115522 | controller | skipping: Conditional result was False 2026-03-20 16:38:45.137367 | crc | skipping: Conditional result was False 2026-03-20 16:38:45.205635 | 2026-03-20 16:38:45.205784 | LOOP [fetch-output : Ensure local output dirs] 2026-03-20 16:38:45.559342 | controller -> localhost | ok: "/var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/work/logs/controller" 2026-03-20 16:38:45.604632 | crc -> localhost | ok: "/var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/work/logs/crc" 2026-03-20 16:38:45.822348 | crc -> localhost | ok: "/var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/work/artifacts" 2026-03-20 16:38:45.825096 | controller -> localhost | ok: "/var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/work/artifacts" 2026-03-20 16:38:46.044104 | controller -> localhost | ok: "/var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/work/docs" 2026-03-20 16:38:46.054047 | crc -> localhost | ok: "/var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/work/docs" 2026-03-20 16:38:46.054124 | crc -> localhost | ok: All items complete 2026-03-20 16:38:46.054157 | 2026-03-20 16:38:46.086607 | 2026-03-20 16:38:46.086797 | LOOP [fetch-output : Collect logs, artifacts and docs] 2026-03-20 16:38:46.858164 | crc | changed: 2026-03-20 16:38:46.858391 | crc | .d..t...... ./ 2026-03-20 16:38:46.858421 | crc | >f+++++++++ crc-cloud-workdir-crc-all-logs.tar.gz 2026-03-20 16:38:46.858441 | crc | >f+++++++++ zuul-output-crc-all-logs.tar.gz 2026-03-20 16:38:46.858459 | crc | cd+++++++++ crc-cloud/ 2026-03-20 16:38:47.322599 | controller | changed: 2026-03-20 16:38:47.322733 | controller | .d..t...... ./ 2026-03-20 16:38:47.322760 | controller | >f+++++++++ zuul-output-controller-all-logs.tar.gz 2026-03-20 16:38:47.441211 | crc | ok 2026-03-20 16:38:47.815281 | controller | ok 2026-03-20 16:38:47.950625 | crc | ok 2026-03-20 16:38:48.297345 | controller | ok 2026-03-20 16:38:48.297539 | controller | changed: All items complete 2026-03-20 16:38:48.297569 | 2026-03-20 16:38:48.394169 | 2026-03-20 16:38:48.394305 | TASK [include_role : fetch-output-openshift] 2026-03-20 16:38:48.424984 | controller | skipping: Conditional result was False 2026-03-20 16:38:48.435774 | crc | skipping: Conditional result was False 2026-03-20 16:38:48.462420 | 2026-03-20 16:38:48.462497 | LOOP [merge-output-to-logs : Move artifacts and docs to logs dir] 2026-03-20 16:38:48.905697 | controller -> localhost | ok: Item: artifacts Runtime: 0:00:00.013941 2026-03-20 16:38:49.147363 | controller -> localhost | ok: Item: docs Runtime: 0:00:00.013757 2026-03-20 16:38:49.364352 | 2026-03-20 16:38:49.364466 | PLAY [all] 2026-03-20 16:38:49.452958 | 2026-03-20 16:38:49.453083 | TASK [remove-build-sshkey : Remove the build SSH key from all nodes] 2026-03-20 16:38:50.128680 | controller | changed 2026-03-20 16:38:50.321723 | crc | changed 2026-03-20 16:38:50.498999 | 2026-03-20 16:38:50.499107 | PLAY RECAP 2026-03-20 16:38:50.499147 | controller | ok: 5 changed: 3 unreachable: 0 failed: 0 skipped: 2 rescued: 0 ignored: 0 2026-03-20 16:38:50.499173 | crc | ok: 4 changed: 2 unreachable: 0 failed: 0 skipped: 2 rescued: 0 ignored: 0 2026-03-20 16:38:50.499191 | 2026-03-20 16:38:50.602259 | POST-RUN END RESULT_NORMAL: [trusted : review.rdoproject.org/config/playbooks/base-minimal/post-ssh.yaml@master] 2026-03-20 16:38:50.615634 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/base-minimal/post-logs.yaml@master] 2026-03-20 16:38:51.288941 | 2026-03-20 16:38:51.289053 | PLAY [localhost] 2026-03-20 16:38:51.306851 | 2026-03-20 16:38:51.306932 | TASK [generate-zuul-manifest : Generate Zuul manifest] 2026-03-20 16:38:51.920049 | localhost | changed 2026-03-20 16:38:51.926626 | 2026-03-20 16:38:51.926757 | TASK [generate-zuul-manifest : Return Zuul manifest URL to Zuul] 2026-03-20 16:38:51.954326 | localhost | ok 2026-03-20 16:38:51.962522 | 2026-03-20 16:38:51.962588 | TASK [add-fileserver : Create SSH private key tempfile] 2026-03-20 16:38:52.314740 | localhost | changed 2026-03-20 16:38:52.319605 | 2026-03-20 16:38:52.319690 | TASK [add-fileserver : Create SSH private key from secret] 2026-03-20 16:38:53.130947 | localhost | changed 2026-03-20 16:38:53.136750 | 2026-03-20 16:38:53.136833 | TASK [add-fileserver : Add fileserver ssh key] 2026-03-20 16:38:53.534445 | localhost | Identity added: /var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/work/tmp/ansible.0im3jhdg (/var/lib/zuul/builds/7ffee4ddf4ea401db94c9ba86e43148b/work/tmp/ansible.0im3jhdg) 2026-03-20 16:38:53.534627 | localhost | ok: Runtime: 0:00:00.010005 2026-03-20 16:38:53.538931 | 2026-03-20 16:38:53.538996 | TASK [add-fileserver : Remove SSH private key from disk] 2026-03-20 16:38:53.838838 | localhost | ok: Runtime: 0:00:00.008597 2026-03-20 16:38:53.878462 | 2026-03-20 16:38:53.878604 | TASK [add-fileserver : Add fileserver to inventory] 2026-03-20 16:38:53.934745 | localhost | changed 2026-03-20 16:38:53.940953 | 2026-03-20 16:38:53.941056 | TASK [add-fileserver : Add fileserver server to known hosts] 2026-03-20 16:38:54.335155 | localhost | changed 2026-03-20 16:38:54.373206 | 2026-03-20 16:38:54.373328 | PLAY [localhost] 2026-03-20 16:38:54.390364 | 2026-03-20 16:38:54.390454 | TASK [Generate bulk log download script] 2026-03-20 16:38:54.413790 | localhost | ok 2026-03-20 16:38:54.431728 | 2026-03-20 16:38:54.431815 | TASK [local-log-download : Check API endpoint is defined] 2026-03-20 16:38:54.481958 | localhost | ok: All assertions passed 2026-03-20 16:38:54.488125 | 2026-03-20 16:38:54.488209 | TASK [local-log-download : Create download script] 2026-03-20 16:38:54.927357 | localhost -> localhost | changed 2026-03-20 16:38:54.936267 | 2026-03-20 16:38:54.936330 | TASK [Register quick-download link] 2026-03-20 16:38:54.971142 | localhost | ok 2026-03-20 16:38:54.998868 | 2026-03-20 16:38:54.998959 | PLAY [logserver.rdoproject.org] 2026-03-20 16:38:55.013817 | 2026-03-20 16:38:55.013880 | TASK [Set zuul-log-path fact] 2026-03-20 16:38:55.030325 | logserver.rdoproject.org | ok 2026-03-20 16:38:55.039812 | 2026-03-20 16:38:55.039877 | TASK [set-zuul-log-path-fact : Set log path for a build] 2026-03-20 16:38:55.065444 | logserver.rdoproject.org | ok 2026-03-20 16:38:55.070860 | 2026-03-20 16:38:55.070924 | TASK [upload-logs : Create log directories] 2026-03-20 16:38:55.977481 | logserver.rdoproject.org | changed 2026-03-20 16:38:55.981964 | 2026-03-20 16:38:55.982031 | TASK [upload-logs : Ensure logs are readable before uploading] 2026-03-20 16:38:56.377465 | logserver.rdoproject.org -> localhost | ok: Runtime: 0:00:00.055741 2026-03-20 16:38:56.387102 | 2026-03-20 16:38:56.387240 | TASK [upload-logs : Upload logs to log server] 2026-03-20 16:39:02.280595 | logserver.rdoproject.org | Output suppressed because no_log was given 2026-03-20 16:39:02.284595 | 2026-03-20 16:39:02.284708 | LOOP [upload-logs : Compress console log and json output] 2026-03-20 16:39:02.365561 | logserver.rdoproject.org | skipping: Conditional result was False 2026-03-20 16:39:02.370942 | logserver.rdoproject.org | skipping: Conditional result was False 2026-03-20 16:39:02.384628 | 2026-03-20 16:39:02.384713 | LOOP [upload-logs : Upload compressed console log and json output] 2026-03-20 16:39:02.430605 | logserver.rdoproject.org | skipping: Conditional result was False 2026-03-20 16:39:02.430856 | 2026-03-20 16:39:02.435275 | logserver.rdoproject.org | skipping: Conditional result was False 2026-03-20 16:39:02.442239 | 2026-03-20 16:39:02.442306 | LOOP [upload-logs : Upload console log and json output]